Improving your code with Static Analysis tools

Update, October 2012. I’ve edited this to include more tools, prices and our experience with each vendor we’ve interacted with.

At Software Verification we create tools for the dynamic analysis of software – analysing your software as it executes. This is an activity that takes places after you’ve compiled your software. There is a complementary method of analysing your software – static analysis. Static analysis is the process of analysing your software before your compile it.

There is a small overlap between the types of bugs that static analysis and dynamic analysis can find. Static analysis will find some of the bugs dynamic analysis finds, but static analysis will also find bugs that dynamic analysis cannot find and dynamic analysis will find bugs that static analysis can’t find. In summary you should have both static analysis tools and dynamic analysis tools in your software tool box.

Tools

If you look around the web you will find a variety of tools, some open source and some commercial. The commercial tools range from the sensibly priced Gimpel PC Lint to tools that are so expensive you need to be sending rockets into space to afford them. For these products where the vendor does not list the price on the website, we have listed POA in the table below. POA means “Price on Application”, which you can think of as code for “very expensive”.

One other thing, most of these POA tools only license you for using the tool for one year. After that you have to buy the software again (not maintenance, but the right to use the software again). So they are even more expensive than you think.

This is not an exhaustive list of tools. There are many more, especially if you look outside of C/C++.

Tool	Type	Cost	Web
Coverity	Yearly subscription	POA	https://www.coverity.com/
Klocwork	Yearly subscription	POA	https://www.klocwork.com/
Mathworks Polyspace	Permanent?	POA, £20,000/user	https://www.mathworks.co.uk/products/polyspace/
Understand	Permanent	POA, 1003 Euro, Maintenance 18%	https://www.scitools.com
CodeSonar	Yearly subscription	POA	https://www.grammatech.com
Pattern Insight	Yearly subscription	POA	https://patterninsight.com/products/code-assurance
Imagix	Yearly subscription	POA	https://www.imagix.com/a/source-code-analysis.html
CheckMarx	Yearly subscription	POA	https://www.checkmarx.com
Parasoft C++ Test	Commercial	POA	https://www.parasoft.com/jsp/products/cpptest.jsp
PVS-Studio	Commercial	3500 Euro, maintenance 80%, minimum 5 users	https://www.viva64.com/en/pvs-studio/
QA C++	Commercial	POA	https://www.programmingresearch.com/qacpp_main.html
SEntry	Yearly subscription	$4,995 US	https://www.vigilantsw.com/
Gimpel PC Lint	Commercial, permanent license	$389 US	https://www.gimpel.com/html/pcl.htm
Microsoft PREfast	Commercial	Free	https://msdn.microsoft.com/en-us/windows/hardware/gg487345.aspx
CppCheck	Open Source	Free	https://sourceforge.net/apps/mediawiki/cppcheck
Splint	Open Source	Free	https://lclint.cs.virginia.edu/
Google cpplint.py	Open Source	Free	https://github.com/cpplint/cpplint
Inspirel Vera++	Commercial	Free	https://www.inspirel.com/vera/

In addition to the above tools there is also a Visual Studio addin (Visual Lint by Riverblade) that can manage many of the above tools and provide you with a very usable, nice interface to the mountains of data some of these tools can generate. Visual Lint is not actually a lint or static analysis tool. Visual Lint’s job is to organise the output of the static analysis tools.

Riverblade Visual Lint

Commercial, permanent license

$399 US (varies)

https://www.riverblade.co.uk/

Our static analysis history

Several years ago we thought getting a static analysis tool would be a good idea. We approached several tool vendors and drew a blank. The prices were astronomical – Coverity (based on their standard pricing) would want $5.6 million (for a one year, single user license) if I remember correctly. An email exchange with them revealed they would provide a discount but the price was still more than half a million dollars. We approached Klocwork but their reseller was only interested in selling us a five user license when we only wanted a single user license. I think that was $20,000 for a one year license.

Recently we’ve approached more vendors:

PVS-Studio

PVS-Studio insist on selling a minimum order of 5 users, which isn’t very useful when you only have a requirement for one user. But you can evaluate without talking to anyone, just download an evaluation from their website. PVS-Studio also works out of the box with your Visual Studio projects. By far the easiest to setup.

Mathworks

Mathworks make you complete a long account form on the website (which appears to be redundant) then wait for an email, then a telephone call followed by two more emails and er, we’re still waiting. Strikes me as a rather good way to lose sales. We’ve evaluated several other tools just while we waiting to even start with MathWorks. Mathworks is at the expensive end of the pricing scale, but by no means the most expensive. But if you can’t even evaluate, how do you get to purchase?

SciTool

SciTools Understand, like PVS-STudio, allow you to download an evaluation. SciTools appears to be hard to setup – it defeated us.

Imagix 4D

Imagix 4D, like PVS-STudio, allow you to download an interactive demo and then to request an evaluation if you like the demo. From the demo we couldn’t see how this is a static analysis tool, so we passed on the evaluation. It seems to be more of a code understanding tool.

Grammatech CodeSonar

Grammatech require you to fill in a form, then they telephone you to start the evaluation.

Checkmarx

We haven’t got around to evaluating yet but their email response time and quality of answers to questions is excellent.

VSLint

We had previously used Gimpel PC Lint, but found it very hard to configure. Then we found Riverblade’s Visual Lint which is an addon for Microsoft Visual Studio. Visual Lint works with many other analysis tools and provides a nice visual front end for viewing the output of the other tools. For us this was perfect as it worked with Gimpel PC Lint and configured it correctly out of the box. No messing with Gimpel PC Lint to make it work with our Visual Studio builds. We’ve had a few problems (our setup seems unusual) and have found Riverblade very responsive to support requests. We now have x86 and x64 static analysis setup using VSLint.

We have been analysing our software using Visual Lint recently. We haven’t found a lot of bugs (great!), but it has revealed subtle signed/unsigned conversions, some logic errors, redundant code and some code that could (in the right conditions) manifest as bugs. It also reports various issues that you may regard as coding style issues. One of the coding styles I hate is the call a function, assign its return value and test a conditional all in one.

    if (!(fResult = ETPhoneHome()))
    {
        RideBicycleIntoSpace();
    }

How much clearer to write:

    fResult = ETPhoneHome();
    if (!fResult)
    {
        RideBicycleIntoSpace();
    }

The compiler will generate the same code, but the second is easier to read and should you need to debug it, the second version is easier to debug. Disk space is cheap. There is no reason to code in the style of the first version. As such the informational warnings like this can be used a mini-style guides. Fix the issue to remove the informational warning from the tool output.

For any warning you don’t agree with or don’t care about you can filter them out.

I must say that I’m very pleased with the outcome of this exercise. I can recommend the combination of Gimpel PC Lint and Visual Lint. Finding these bugs so easily has paid for the software tools used to find them.

Unusual benefits

I’m going to list a few things the PC-Lint/Visual Lint combination warns about which I’ve found beneficial.

Variables not initialised in constructors and functions.

Easy mistakes to make. Often caused by logic errors rather than forgetfulness.

Unused variables, unused functions and unreachable code.

Variables you once needed, left behind after edits, cluttering the place up. Same for debugging function and historically obsolete code. Useful to know what you can safely remove. A word of caution though – be sure to check for conditional compilation excluding the code for the lint analysis. Lint can get this wrong sometimes.

Incorrectly defined copy constructors and assignment operators.

I’ve found these to be a boon. The previous constructors and operators worked OK, but better to have them in the style that STL will expect.

Lack of const

Before:

void myFunc(char *title);

void myFuncRef(CString &title);

After:

void myFunc(const char *title);

void myFuncRef(const CString &title);

These are minor wins as this change only gets suggested if Lint can determine that the parameter is not modified. However it is a win for the rest of your software as you can now pass const objects into these functions without casting them to none const (ouch!) and also know that the function, if modified to modify the objects will not compile properly. So it is a useful maintenance win.

Change from pass by value to pass by const reference.

Before:

void myFuncRef(CString value);

After:

void myFuncRef(const CString &value);

This is a major win. You get the maintenance wins from the previous example plus you get a CPU saving because there is no implicit object copying happening to pass the value into the function. In the case of complex objects (like CString) which allocate memory on construction and destroy memory on destruction there you save even more CPU cycles because the memory manager is not called. Additionally you are less likely to fragment the heap because the heap is used less. Win, win, win!

You definitely want to be making changes like this. Visual Lint will tell you stuff like this in its results.

The only downside to these changes are that you need to rebuild after the changes. If the changes are in a core header file or a class that many files use you’ll spend a while rebuilding. But this pain is short-lived. You’re making the software more robust in the process and long term that is a worthwhile trade off.

Pricing

I think most static analysis tool vendors are missing a huge opportunity. By pricing themselves so expensively most software houses will not purchase such tools. Then to compound matters they make it a yearly subscription so you have to spend the full price again each year. When I look at purchasing software the first thing I look for is a published price list. The lack of a price list sends the price signal “we are expensive”.

This is a mistake – surely the message should be “we offer value” and be confident enough to state your pricing.

I can’t help thinking hidden pricing means the price varies depending on what they think you can pay. That’s really not a good message to send. Since the previous version of this article I’ve approached some of the POA vendors and found that some of them are expensive and some are very affordable, but still have unlisted prices “just because that is the industry convention”.

Any company that chooses to price their tools so they are affordable to software developers (in the $200 – $1000) range will have a much better opportunity for these tools.

From what I can see only Gimpel and Riverblade have taken this route.

SciTools Understand is a sensible price, but hard to setup.

PVS-Studio come close but lose it with their minimum user limit of 5 users. Who is going to pay for non-existent users?

Conclusion

If you are not using static analysis tools I recommend that you spend some time investigating such tools. The free tools have their pitfalls – incomplete implementation and they don’t always have support. CppCheck seems to be highly thought of (it doesn’t work properly with our code). If you are using Gimpel PC Lint or want to use a lint then I recommend Visual Lint for use with Visual Studio. If you have the budget then check out the POA tools in the table above.

Some people make the mistake of thinking that static analysis tools replace dynamic analysis tools (like C++ Memory Validator, etc). Static analysis tools complement dynamic analysis tools. There is a small middle ground where some bugs will be identified by both a static analysis tool and a dynamic analysis tool, but each group of tools can identify whole classes of bug that the other tool cannot identify.

If you are a tool vendor listed above please let us know your pricing and/or licensing terms if this article lists it incorrectly.