We will attempt to answer any of your questions in this section of the site. If there are any other questions you would like to put to us please contact us and we will get back to you with a prompt answer.
No.
Memory Validator inspects all Win32 heaps (HeapCreate etc), the GlobalAlloc() heap, LocalAlloc() heap, CoTaskMemAlloc() heap, IMallocSpy, as well as the C runtime heap(CRT). If you are using any of these heaps Memory Validator is capable of monitoring the memory allocations and deallocations.
If you are using a custom heap manager of your own design (or a 3rd party design) you can provide Memory Validator with information via the API so that Memory Validator can track the behaviour of your custom heap manager.
This question in not straightforward to answer because the answer depends in part, upon the target program that Memory Validator is examining. A general answer will be given followed by some qualifications to explain the answers. These answers are based upon the beta testing on programs varying in size between 10,000, and 2,000,000 lines of C++.
We have not provided % figures as we believe such figures are misleading. The performance change will be always in relation to the data generated by the target program (for example a 2,000,000 line CAD program we have tested starts up in much less time than a 300,000 line web authoring program we tested. The reason for this was not coding quality, but the nature of the work each of the programs was doing during startup. The web authoring program was preparing a lot of data during startup which the CAD program did not need to do (*)).
Least impact listed first.
In general the worst hit is with the buffer overrun detection. However if you program uses the C (and Win32 shell) string functions very little there will be no noticeable performance change. We have only found one program where the buffer overrun detection caused the program to run very slowly, and this was only during program startup when a very large amount of string processing was being done. After the program startup the program performed as normal. The buffer overrun detection is probably the most infrequently used feature on Memory Validator.
The handle tracking functions produce very little performance overhead even on very large programs.
The memory allocation tracking has a low overhead unless there are large numbers of allocations in very tight loops.
The uninitialised data detection, by its very nature can have a high overhead, but this is dependent upon the data it is examining.
(*) On a 2,000,000 line CAD program with all options enabled, Memory Validator launches and runs the target program in 90 seconds. With the uninitialised data and buffer tracking disabled Memory Validator launches and runs in 30 seconds. One of Memory Validator’s competitors takes over 40 minutes to do this task and often fails towards the end of the 40 minute period.
Note that for the web authoring program mentioned above, startup times is typically 21 seconds. If uninitialised data detection is enabled, startup time increases to 47 seconds. If buffer overflow detection is enabled, startup time increases to 15 minutes. This is quite unusual and is due to the large amount of string manipulation the program does at startup whilst it sets up various data structures which later on in the program will improve its performance greatly.
If in doubt about the performance impact of Memory Validator on your product we advise you to try Memory Validator on your product to determine the performance impact. We think you will be favourably impressed compared to our competitors.
Memory Validator detects uninitialised memory in memory allocated by the debug C runtime heap. The debug C runtime heap initialises all allocated memory with a signature byte of 0xCD. It also initialises any uninitialised stack variables with a signature byte of 0xCC. Memory Validator hooks all constructors of C++ objects and at the end of the constructor examines the object for any uninitialised data bytes. If any are found they are reported to the user.
0xCD is a valid value for a data byte but is unusual to be found as a WORD as 0xCDCD or a DWORD as 0xCDCDCDCD.
0xCC is a valid value for a data byte but is unusual to be found as a WORD as 0xCCCC or a DWORD as 0xCCCCCCCC.
It is up to the user to determine if the report of uninitialised data is correct or not.
The technology for hooking the object’s constructors is the same as is used to hook COM objects.
Memory Validator hooks COM objects by hooking the Win32 functions that return COM objects. These objects are then queried for their QueryInterface, AddRef and Release function pointers. These functions are then hooked by rewriting the instruction stream using some proprietary code. The hooks then examine the return values to detect the reference count or returned object. Note that because rewriting the instruction stream is not a generally recommended practice this can sometimes fail (typically when the compiler optimises two functions to share some common code). Memory Validator tries to detect when hooking will fail and refuses to hook any functions that it knows it cannot hook safely. However some COM objects cannot be hooked safely – we have found that you can debug some COM objects using Memory Validator, and not others because of the internal structure of the COM objects.
Memory Validator provides some controls for the COM hooking procedure. See the section entitled Instruction Stream Rewriting Hooks on the tab of the dialog. The instruction stream rewriting concept has been around for years (early video games in the 1980s for the Commodore VIC 20 and Commodore 64 often rewrote their instructions on the fly to gain a speed advantage) and is used, in varying forms, by some of the competing products and complimentary products to Memory Validator, and by Microsoft®.
Memory Validator can be configured to track as much or as little as you wish. If you are only interested in the handles returned from GetDC() and passed into ReleaseDC() you can configure Memory Validator to do that. If on the other hand you want to know about every handle allocated and deallocated, every memory allocation and deallocation and all COM objects, all memory errors and so forth, Memory Validator can do that too. When Memory Validator is tracking more data items it has more work to do and Memory Validator will run slower when tracking all data items than when tracking only a few.
Memory Validator was beta tested on a 2,000,000 line C++ CAD program and performed very well even when tracking all data items.
Memory Validator provides many configuration options that control the collection of data and the display of the data. It is possible to configure Memory Validator to ignore the data you want collected.
For example: You may have run a session with Memory Validator and turned off all the handle collecting functions from the Data Hooks tab on the Data Collection dialog. If you do this, even with the hooks inserted into your target program the hooks are disabled. No information on handles will be communicated to Memory Validator.
Possible reasons for the data you require not being collected:
Data collection. Data collection is turned off. Check that the green data collect icon is not displayed on the session toolbar (it will be disabled if data is being collected).
Data collection hook groups not installed. The appropriate hook group for the data you wish to view may not have been installed in the target program when the target program was attached to. Enable the hook group and re-run your session. The hook groups installed are configured from the Collect tab on the Data Collection dialog.
The individual hooks for the function you are interested in may have been disabled. Check the hooks using the Data Hooks tab on the Data Collection dialog. If the data hooks were installed, then the hooks will start collecting data once you enable the data hooks.
Data is being collected but not displayed. Check that the controls for the view you are using is set so that the appropriate data will be displayed. Some of the views have the data display controls as part of the view. The Memory tab has its control on the Display tab of the Display Settings dialog.
Check that there are no filters (in Global, Session and Local filter groups) that could be suppressing the display of the data.
Check that the thread filter is not suppressing the display of the data.
Assuming the crash is not a crash you would normally get when not using Memory Validator, there are five possible causes:
If you have the COM Reference Counting hooks installed, this may be a cause of the problem as these hooks rewrite the instruction stream. Sometimes rewriting the instruction stream doesn’t work because some object code produced by the optimising compiler, or hand written assembly code shares common routines. Try turning off the COM Reference Counting hooks and trying your test again.
If you have the Uninitialised Data hooks installed, this may be a cause of the problem as these hooks rewrite the instruction stream. Sometimes rewriting the instruction stream doesn’t work because some object code produced by the optimising compiler, or hand written assembly code shares common routines. Try turning off the Uninitialised Data hooks and trying your test again.
Try removing all Target DLL extensions and Validator DLL extensions and trying your test again.
Some DLLs from third party vendors which use system wide hooks do not interact with Memory Validator and the target program very well. If you can identify such DLLs, add then DLL name to the Don’t Hook tab on the Advanced Settings dialog and try again. Memory Validator will not hook these DLLs.
There may be a bug in Memory Validator. Whilst we have tried to make Memory Validator as robust as possible there are scenarios that we may not have thought of and thus possible errors we have not catered for. Try to ensure that the crash never happens if you are not using Memory Validator.
If the crash still happens and you have tried all the suggestions on this page, please send details of the error to Software Verify and we will try to reproduce the crash with a view to fixing any bugs found in as timely a manner as possible. Please contact us at the address provided on the contact page.
If you have specified a callstack depth for all data items collected the collected callstack may be shorter for a variety of reasons.
The actual callstack may have had less entries in it that the specified depth.
The callstack had more items in it that the specified depth, but some of the items in the callstack were omitted to provide a stack trace that was useful. An example will shed some light on this rather bizarre statement. Consider for example tracking a call to operator new. For Debug builds operator new calls to malloc_dbg(), thus the call stack will be something like (various functions omitted for clarity):
Memory Validator’s callstack in this instance would be:
so that examining the source code to myFunctionThatCallsNew() would show the call to operator new(). This is done so that program calls to operator new() are not confused with program calls to malloc() or malloc_dbg().
In the example above Memory Validator has removed 2 items from the callstack. Since Memory Validator only collected your specified depth, you can see why the number of items in the callstack may be slightly less than you expect. Memory Validator attempts to correct for this by estimating the number of functions that should not be included in the trace prior to collecting the trace. Most of the time Memory Validator estimates correctly. On the occasions Memory Validator gets its estimates incorrect the stack trace will be shorter than expected.
If for a particular bug you are investigating the stack trace is a bit shorter than you need, modify the data collection settings so that you collect more of the callstack, or collect the entire callstack.
We have provided hooks for all the functions that we think are required to be hooked in order to provide as comprehensive a tool to aid you in your software development tasks. As times pass Microsoft® releases new APIs with new functions in the APIs. We intend to modify Memory Validator as new APIs are released.
However if we have missed an API which you think is important for a bug you are working on, please let us know, and if we think the API should be added to the list of APIs that Memory Validator hooks, we will do so. Please contact us at the address provided on the contact page.
We have tried to add as many features to Memory Validator that we thought would be useful to the potential users of Memory Validator. In fact every feature in Memory Validator has been used to solve problems and bugs on consulting work carried out for our customers and on internal projects at Software Verify Limited. We know the features we have put in the product are useful.
However it is possible we have overlooked a feature that you may find very useful and which you cannot work out how to add to Memory Validator via an extension DLL. We will be pleased to consider all ideas for new features to Memory Validator. Please contact us at the address provided on the contact page.
A Quake III mode, however much fun, is not useful in solving bugs. Sorry guys!
Memory Validator stores most of the configuration data it needs in the registry. However some data, such as the data hooks, coverage data and filter data is stored in files. This section describes the file extensions used by Memory Validator so that you can recognise such files. The file data structure is not described as it may change from version to version of the product.
Session Export and Session Save
html. HTML export files
xml. XML export files
mvm. Memory Validator session files
Settings, Filters, Coverage
mvs. Settings
mvf. Filter files
mvc. Coverage files
mvd. Coverage files
Ordinal To Symbol Conversion
def. Linker definition files
ord. Ordinal data files
Program Launch, Extensions
dll. Extension DLLs
exe. Program files
Source Code
cpp. C++
c. C
h. C and C++
cxx. C++
hxx. C++
hpp. C++
A common reason for not seeing data that you believe you should see is that Memory Validator has been configured to disable the hooks required to collect the data you want. Check the following criteria:
– Check the correct hooks to monitor the memory, handle and/or COM allocations are installed. See the Collect tab of the data collection settings dialog.
– Check that the hooks are enabled. See the hooks tab of the data collection settings dialog.
– Check that the collected data is displayed. See the display tab of the data collection settings dialog.
– Check that global and session filters are not filtering the data, thus preventing its display.
– Check that local filters (for the view being used) are not filtering the data, thus preventing its display.
– Check that the target program really is executing the code you think it is executing.
When Memory Validator cannot locate debug information for DLLs that have their functions exported as ordinal values, the functions are named ‘Ordinal’ with the decimal number of the function following. The ordinal values can be displayed as function names if Memory Validator knows which linker definition (.def) file refers to each DLL having ordinal exports. You can define ordinal handling behaviour using the Ordinal Handling features of Memory Validator.
Also, don’t forget to select the Map Ordinals to function names check box on the Source tab on display settings dialog.
If you have used the features of Memory Validator and the ordinal names have not been converted in symbol names, there are two possible reasons:
– You have not defined the ordinal to symbol conversion for the DLL in which the ordinals are defined.
– You have forgotten to enable ordinal handling. Using the Source tab on Display Settings dialog, select the Map Ordinals to function names check box.
Yes, Memory Validator can hook delay loaded functions. Various controls are provided so that you can control:
If delay loaded DLLs are hooked. See the Don’t Hook tab of the Advanced Settings dialog.
If delay loaded functions are hooked. See the Collect tab of the Data Collection Settings dialog.
There are many reasons why your program may run slowly when used with Memory Validator.
Memory Validator provides you with many options to enable you to turn off data collection for data that is not important. Turning off unwanted options prevents Memory Validator from spending time examining data you don’t want collected.
If you are not trying to isolate memory corruptions, turn off the buffer checking option.
If you are not trying to isolate uninitialised data, turn off uninitialised data detection.
If you not trying to detect handle leaks, turn off all handle relatedhooks.
If you are not trying to detect leaks in GlobalAlloc, LocalAlloc and HeapAlloc, turn off these options.
If you are not trying to detect CRT leaks, turn CRT leaks off.
If you do not need to collect complete callstacks, collect only the part of the callstack that is interesting to you. Depending how deep your programs callstacks get, changing this option can have quite a dramatic impact on your program’s performance.
If your program is still running slowly, it may be because the program is allocating many blocks of memory in a tight loop. When this happens, Memory Validator gets swamped with the sheer volume of data it needs to track, and symbols (for the callstack) to resolve. When the program exits the tight loop, the program performance will return to more normal speeds. Often this is a sign that the program being worked on could be improved by redesigning its memory allocation strategy. Examining the statistics on the objects view will give you an insight into the number of allocations your program is making and the allocation size frequency of those allocations.
Some double delete callstacks do not show the allocation location. There are four possible causes for this.
char *ptr;
ptr = new char [3000];/ .. do some work
ptr = ptr + 10; / pointer no longer points at the start of the memory block
delete [] ptr; / incorrect
/ .. do some work
delete [] ptr; / incorrect double delete
In the above code fragment, not only is the delete call incorrect because of the incorrect pointer value, there is a second call to delete with the same pointer value. When Memory Validator looks for the associated allocation location, it will never find it because there are no allocations at the address “ptr + 10”. This is why although the double delete can be detected, the allocation location cannot be detected.
When using the Find Memory Dialog or the Find Object Type dialogs it is possible to see datatypes to use for a search that give no search results. The reason for this is that the datatypes have been loaded from the file type cache. To remove these datatypes from the cache you must flush the datatype cache.
The garbage collection function is provided so that you can reset the leaked memory level to zero whilst doing a debugging test. For example, you may do a test to determine the memory leak exists, and then form a theory as to why that happened. Based on this you may want to remove these leaks prior to retesting with you knowledge of the leak. The garbage collector is not provided to replace the malloc/new allocators. If you require a garbage collection solution there are several vendors of garbage collectors to choose from. We do not offer opinions as to which garbage collection solution to use. We also do not recommend garbage collection as a solution as it always has poor performance when your application consumes as much memory as the machine your application is running on. At this point each pass of the garbage collector will touch all pages and cause very heavy virtual memory paging effects with the consequence performance hit.
The garbage collector never collects internal MFC objects, which are temporary and are managed by MFC. These do not normally show up as leaks. The Microsoft® C runtime has various leaks in it, or apparent leaks in it. Some of this memory is referenced by thread local storage slots and will be cleaned up when the particular thread is destroyed. Any memory which is not marked as _IGNORE_BLOCK by MFC or _CRT_BLOCK by the C runtime can and will be garbage collected. Memory on Win32 heaps will not be garbage collected.
A potentially leaked memory block is a memory block that has pointers to parts of the memory block, but not to the start of the memory block. This can happen because the memory is deliberately managed this way (CString for example uses offsets to help it manage various aspects of CString). This can also happen because a stray pointer elsewhere in memory that is managed by the program is pointing into the block. The memory is marked as potentially leaked because it may be a genuine leak or it may be a deliberate strategy of the program. It is up to you, with your knowledge of your program to determine if it is a genuine leak or not.
This usually means that the object is being pointed to by a pointer held in a static object, and object on the stack or an object in a Win32 heap created using HeapCreate().
Unknown pointers are usually pointers found in static program data or Win32 heap allocation data. Such memory maybe a static C++ object or a C struct, or just an array holding pointers. The ‘Leak’ field shows ‘No’ because the pointer has not been allocated.
No Memory Validator does not require you to recompile and/or relink your program. Memory Validator does also allow you to link it into your program if you wish to access the extensible features of Memory Validator.
For the target application:
For the user interface: