Code coverage with VS Test and Coverage Validator

In this blog post, I’m going to give you an example of running .Net Core unit tests with VS Test (formerly MS Test) and Coverage Validator. It’s the same process for regular .Net and C++.

First, let’s discuss the program we’re going to launch and the program we’re going to monitor.

vstest.console.exe

VS Tests are run by vstest.console.exe. So that’s the program we’re going to launch. On my machine the path is:

C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe

In this example, I’m showing VS 2019 community edition, but it doesn’t matter which VS or version you use.

vstest.console.exe is not a .Net Core application (it’s a regular .Net application). You can check this with our free tool PE File Browser.

testhost.exe

vstest.console.exe executes the tests by running testhost.exe. We need to identify which testhost.exe to run (there will be several installed on your machine) and then configure Coverage Validator to monitor that testhost.exe when vstest.console.exe is run.

We haven’t worked out a way of identifying which testhost.exe VS Test is going to use, but once you’ve found it that will be the one forever.

On my machine testhost.exe is in c:\users\stephen\.nuget\packages\microsoft.testplatform.testhost\16.5.0\build\netcoreapp2.1\x64\testhost.exe

Note that despite the path testhost.exe itself is not a .Net Core application (it’s a regular .Net application).

Video or step-by-step

I’ve created a video showing you how to configure Coverage Validator, but if you prefer step-by-step instructions, these are listed below the video in this blog post.

Coverage Validator

To get started we need to launch vstest.console.exe to run the tests.

Click the Rocket icon on the toolbar. This will display the Launch Application or Service dialog.

Choose Launch Native and .Net applications.

Although we’re going to monitor code coverage in a .Net Core DLL, the application we’re going to launch to do this is not a .Net Core application, so we’ll use the regular .Net and native launcher.

You can also launch using Launch->Applications->Launch Application…, or F4, these will take you straight to the launch dialog/wizard, skipping the previous dialog.

The Start a Native/.Net application dialog is displayed.

Now we have to configure the start application dialog. We’ve got to:

• choose the application to launch
• edit the applications to monitor
• set the application to monitor
• set the arguments for the unit test
• set the startup directory

1. Set the application to launch.

   Next to the Application to Launch field click Browse… and select your vstest.console.exe
   Example: C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe

   

2. Edit the applications to monitor.

   This is a multi-stage process. If the application to monitor has never been configured for the application being launched, you will need to configure the applications that can be set as possible applications that can be monitored. If the application to monitor has been configured, you can edit it to add or remove applications, or you can just use the selection of configured applications. Note that after editing the Application to Launch field the Application to Monitor field will auto-populate itself, choosing a default value for the application to monitor. If this has never been configured before it will choose the same application as the application being launched (in this case vstest.console.exe). If this has been configured it will choose the default application for the configuration.

   For this blog post I’m going to assume that the application has never been configured, and show you how to configure it. Editing is just a subset of these actions and does not need it’s own blog post. If you already have applications to monitor configured you can skip this step. Next to Application to Monitor, click Edit…

   

   • The Applications to Monitor dialog is displayed.

     

   • We need to add an application to the list of applications to monitor. On the Applications to Monitor dialog click Add…

     

     • The Application to Monitor dialog is displayed.

       
       Note that the EXE and DLL are set to represent the current application you are launching. For Native and .Net applications the DLL field will be empty (only set for .Net Core applications). If you wish to edit these values to configure for a different application than the one you are launching you can edit these via the Edit… button.

     • On the Application to Monitor dialog click Add…

       

     • The Application and DLL dialog is displayed.

       

       • On the Application and DLL dialog click Browse… and select your testhost.exeExample: c:\users\stephen\.nuget\packages\microsoft.testplatform.testhost\16.5.0\build\netcoreapp2.1\x64\testhost.exe

         

       • Click OK to accept the EXE and DLL combination.
     • You should now have two entries, one for testhost.exe and one for the testhost.exe with a full path.

       You can repeat the Add… process for any additional applications you wish to configure. Optional: If you want to set this as the default application to monitor choose the appropriate entry in the Default application to monitor combo box.

     • click OK to accept the list of applications to monitor.

     The Applications to Monitor dialog should now show one entry for vstest.console.exe and testhost.exe.

     

   • click OK to accept these definitions of applications to monitor.
3. Set the application to monitor.

   In the Application to Monitor combo select the entry for testhost.exe. We intend to monitor the first testhost.exe that is launched, so set the Launch count to 1.

   

4. Arguments:

   Enter the full path to your DLL to test. Example: E:\om\c\testApps\unitTests\HelloWorldCore\HelloWorldCoreNUnitDotNet5\bin\Debug\net5.0\HelloWorldCoreNUnitDotNet5.dll

   

5. Startup Directory.

   Enter a path for the startup directory. A default will have been set based on the Application to Launch, but for unit test work you’ll need a writeable directory, so you’ll need to edit this value to something appropriate.

   

6. If you’re using the Launch Dialog click Launch.

   
   If you’re using the Launch Wizard click Next until you get to the last page of the wizard then click Start Application.

   

   
   
   

Results

For the test I’ve configured in this blog post, the results show code coverage for the unit tests, for the code tested by the unit tests, for some autogenerated code, and for some code in the test framework.

Some of the results are from code that isn’t your unit tests. You’ll need to filter these results by configuring Coverage Validator to ignore these code locations in future tests.

Filtering Autogenerated Code

To filter out of the autogenerated code, right-click on the entry and choose Instrumentation Filter->Exclude Filename.

Filtered-out entries are shown in grey. Next time you run the instrumentation they won’t be included in the coverage results.

Filtering Test Framework Code

To filter out the test framework code, right-click on the entry and choose Instrumentation Filter->Exclude DLL.

Filtered-out entries are shown in grey. Next time you run the instrumentation they won’t be included in the coverage results.

Any questions?

Hopefully, this blog post has answered your questions about how to get code coverage with VS Test and Coverage Validator.

But you may have other questions. Please let us know at support@softwareverify.com and we’ll be pleased to help you.