Changing how we install our software

With the next release of our software (later today) we are changing how we install our software.

The installers stay the same, the software (as you see it) stays the same, but under the hood, things are different. This article describes that change, why we made that change, the consequences of that change, and the work we did to ensure the software keeps functioning.

History

First, some history.

System32

Once upon a time, software developers routinely placed their DLLs in the Windows system directory, and then when Windows 95 came along, they placed their DLLs in the Windows system32 directory. This eventually resulted in a mess, as far too many DLLs from different software vendors ended up stored in one place.

To solve this Microsoft stated that an application’s DLLs should be stored with the application. Microsoft also made installing their redistributables easier and made various other changes that made “DLL Hell” go away (or manifest in subtly different forms). Over time most software vendors changed how they distributed their software.

Debugging Tools

When we started Software Verify Limited (originally as Software Verification Limited, which no-one could spell!), we noticed that for certain types of problem we needed to investigate, it was quite hard to get the appropriate DLLs to load into the target application unless the DLLs were stored in the system32 directory.

We took the view, rightly or wrongly, that because we were providing tools that do unusual things with applications we could also do non-standard things with how we installed the software. We also added code to the user interface so that each time it started it would copy it’s DLLs to where it thought they needed to be (usually system32, sometimes SysWow64 – the 32 bit system32 on a 64 bit machine).

This behaviour was so ingrained and so core to how the software worked, that even when we spent 3 months rewriting the software many years ago to work nicely in the administrator mode enabled world of Windows Vista/7/8/10, it never occurred to us to change our abuse of the system32 directory.

Anti-Virus

Then one day, a developer researching our tools emailed us saying that he liked our tools but he was constantly fighting his anti-virus tool (BitDefender) that was flagging our tools as viruses because we copied files into the Windows system32 directory. He had disabled this feature but really wanted our software modifying so that we didn’t trigger the Anti-Virus software.

We’d also had reports from other developers that other anti-virus tools had flagged our tools as viruses – which we had assumed was a reference to the fact that our software uses CreateRemoteThread(). The function CreateRemoteThread() can be used for good and for ill. The Toolhelp and PSAPI libraries use this function to query data from running executables. So does Process Explorer from SysInternals (now part of Microsoft). We also use it. So do Bad Guys ™.

These various reports indicated to us that we had a problem, so we started investigating what needed to change to get the many moving parts of our software to work correctly without us touching Windows system32 or Windows sysWow64.

The change to what we install

Things we don’t do any more:

• We no longer install any Software Verify DLLs or EXEs to system32 or sysWow64.
• We no longer copy any Software Verify DLLs or EXEs to system32 or sysWow64 when the user interface runs.

So how do the DLLs get found?

One solution to this would be to place the path to the installed tool in the PATH environment variable. We considered this for a short while. But we have a lot of tools and the paths to them will be long if stored in the default location of c:\program files etc. We didn’t like the idea of someone installing C++ Developer Suite x64 / x86 and suddenly being faced with 8 new, very long paths added to their PATH environment variable. We had to come up with a better way.

We decided that we’d update the PATH environment variable to point to the tool in use, at the time the tool being used launched an application, launched a batch file, attached to a running application, or started working with a service. This way, only the application being monitored has it’s PATH updated, and it only has it’s PATH updated with the path to the tool in use, not all tools that have been installed. It’s elegant. It results in path isolation, so incompatible DLLs for one tool can’t affect another tool. And the user’s PATH environment variable doesn’t get polluted with lots of PATHs to our tools.

OK, so that sounds easy to do. Is it? Not exactly. There are quite a few use cases we need to cover. Some are trivial to implement, other quite involved. For all of the cases listed below we need to be able to handle:

• 32 bit tool working with 32 bit application.
• 64 bit tool working with 32 bit application.
• 64 bit tool working with 64 bit application.

We made a grid of everything we needed to test, in every combination of tool (32 bit / 64 bit) and launch/batch/inject/service (32 bit and 64 bit). It came to 140 separate tests cases, each with it’s own test peculiarities, often requiring new and inventive ways to address the problem. When working through the services section we had to reboot a few times as we managed to get the service manager into a rather confused state :-).

Why does the PATH matter so much?

The reason the PATH matters so much is because when you’re injecting a DLL into another process so that you can gather information (flow tracing, code coverage, memory allocations, profiling, thread analysis) the DLL you’re injecting almost certainly has dependencies on other DLLs. If those dependencies can’t be satisfied by being found on the DLL loader search path, the DLL won’t load.

The reason we used to store the DLLs in system32 (or sysWow64) is because it’s an easy (and lazy) way to get the DLL dependencies found.

However, if we can modify the PATH suitably, we can ensure the DLLs can still be found.

Launching native (unmanaged) applications

This is when we start a typical application built using C, C++, Delphi, Visual Basic etc.

We modify the PATH in the tool prior to launching the target application. The target application inherits the environment variables of the parent process (the tool launching it).

Launching .Net (managed) applications

This is when we start a typical application built using C#, Visual Basic .Net etc.

We modify the PATH in the tool prior to launching the target application. The target application inherits the environment variables of the parent process (the tool launching it).

Launching applications from batch files

This is when we start an application from a batch file.

We modify the PATH in the tool prior to launching the target application. The target application inherits the environment variables of the parent process (the tool launching it).

This method will work for most use cases. If only life were that simple…

Launching applications from batch files with no path

This is when we start an application from a batch file but the creator of the batch file has explicitly modified the PATH environment variable, nullifying any changes we made to the PATH environment variable when we launched the command processor that is executing the batch file.

We need to intercept where the command processor starts the process and then modify the PATH in the environment block passed to the target application.

Launching applications from other applications

This is where we monitor the Nth application launched by another application.

We need to intercept where the first application starts the process we wish to monitor and then modify the PATH in the environment block passed to the target application.

Attaching to running applications

This is where we need to inject our DLL (which is in a known location that is not in the PATH) into the address space of a running process that already has the PATH environment variable set, most likely without including a path to where our DLL (and it’s dependencies) is located. The key is the fact that the injected DLL has dependencies. If the dependencies can’t be found, the DLL load will fail.

We had to invent a way to modify the PATH in the already running application before we inject our DLL with all it’s dependencies.

Working with services

This is where we need to get our DLL loaded by the tool’s NT Service API (each Validator has it’s own NT Service API).

This isn’t as straightforward as it seems. The path to the DLL isn’t known (the user could have installed the Validator anywhere). We have to find a way to get the PATH information into the service so that the various NT Service APIs can load the DLL from the correct location. To do this we also updated all the NT Service APIs with additional functions to help debug any NT Service API failures. A separate blog post covers this topic. Services run in quite locked down environments, getting this information into each service was quite an interesting adventure.

Changes to installer and Validators

The way our software tools are written you can normally install one version over the top of a previous version without needing to uninstall the previous version. For this next release, with the new DLL layout, we recommend uninstalling the previous version. However this is not essential as we have added some code to the licence key installer and also to the main Validator user interface, code that will check for the existence of Software Verify DLLs in the Windows system32 directory. If Software Verify DLLs are found in this directory, you will be informed and offered the ability to remove these DLLs.

We recommend removing the DLLs, as we cannot predict the interaction between DLLs left in the system32 directory and the new version of the Validator software. It is possible that for most, if not all, use cases that the software will function correctly. This depends on the way the Windows DLL Loader searches for DLLs and how it loads DLLs. If the system32 directory DLLs are found first then the wrong DLLs will be loaded. This is why we recommend removing them.

“But if I delete the svl*.* files in system32, won’t my older tools stop working?”

Older versions of our software tools will continue to work. Every time you start the GUI for an older tool, it will copy the DLLs it needs back into system32 so that it can function. Then, the next time you start a modern tool, you’ll get a prompt about the svl*.* files and have the option to delete them. With both solutions, the tools keep working. However, if you have an anti-virus tool (for example, BitDefender) installed, that will then delete those files, preventing the tool from working. That’s why the new mechanism is better, the tools work and anti-virus tools are happy.

The above scenario isn’t ideal, but it will allow older and newer tools to work together. If this is a serious issue for some people we may add an “always delete the files” option so that subsequent starts of the newer tools automatically delete the svl*.* files from system32. If you need this functionality, please let us know.

We believe the current versions of our tools are better than the older versions and that upgrading to the current versions will provide a better experience, both in functionality, better user experience, and better documentation. This is the 2nd time in our company history (nearly 16 years) that we’ve chosen to make a non-backwards compatible change with previous behaviour (excluding changes to saved data file formats).

Summary

We’ve changed how we install our software. We no longer do things that might alarm anti-virus tools. As a user of our software, you shouldn’t notice any change in functionality. Everything should continue working as normal.