Monitoring 3rd party memory allocation APIs

Memory Validator provides built in support for the common and not-so-common Memory Allocation APIs in Win32, C, Delphi and FORTRAN 95. However, suppose you are using a 3rd party product that also has a memory allocation API, how do you monitor the memory allocations or handle allocations in that case?

Third party memory allocators are quite common, you probably have one or more installed on your computer – for example any MAPI compliant application will implement the MAPI memory API in the DLLs installed for the application. Many database and middleware vendors also provide their own memory management layer.

To overcome this hurdle, Memory Validator provides a custom hook facility that allows you to define what each API looks like and tell Memory Validator how to treat the data collected from that API (alloc, realloc or free).

This tutorial demonstrates how to monitor an example memory allocation API.

This is an advanced topic – you must be accurate in your specification of the function definition and calling convention otherwise your application may crash.

Example projects for the example DLL and a test application are included in the installation directory with Memory Validator. Load the project workspace testCustomHooks.dsw. If you test the example custom hooks, ensure that you also track the other allocators in the DLL, otherwise you may get misleading results.

The example API is defined in the DLL testCustomDLL.dll looks like this:

extern void * __stdcall customStdCallAlloc1(DWORD size);   // input param

extern void __stdcall customStdCallReAlloc2(DWORD size,    // input param
                                            void  *ptr1,   // input param
                                            void  **ptr2); // output param

extern void  __stdcall customStdCallFree1(void *ptr);      // input param

Note that this looks like a typical C malloc/realloc/free group of functions. There are two differences:

1. The functions are defined as __stdcall. This means the function cleans up the stack space when the call has completed.
2. The realloc function has been defined so that the input pointer is passed as a pointer to a pointer and the return value is passed back to the caller via a pointer.

We have added these differences to deliberately demonstrate how to address the issues of data being passed into/out-of a function via a pointer.

 

 

Getting the function names

For extern “C” functions the name is the name of the function with no decorations or function name mangling. For C++ functions, the name is the mangled function name. For __stdcall functions the name may or may not be decorated with a leading underscore, trailing @ and decimal size. The easiest and most trouble free way of determining the function name is to use the depends.exe utility that comes with Visual Studio.

Start depends.exe and load the DLL to be hooked into the DLL. Examine the Export Address Table (highlighted) to find the function name(s) in the DLL.

The image below shows the three functions we are interested in, with grey backgrounds.

Thus we can see that the functions we wish to hook have decorated C++ names as follows:

    ?customStdCallAlloc1@@YGPAXK@Z

    ?customStdCallReAlloc2@@YGXKPAXPAPAX@Z

    ?customStdCallFree1@@YGXPAX@Z

Hint: An easy way to get these names without having to type them is to select each name in the Export Address Table of depends.exe, and copy and paste the names into notepad and then use the names in notepad with Memory Validator.

 

 

Specifying the hooks

Display the settings dialog

by clicking on the settings icon. On the settings dialog, select the custom hooks tab.

Adding custom allocation hook

Our example memory allocation function is

    extern void * __stdcall customStdCallAlloc1(DWORD size);    // input param

and has a mangled name of

    ?customStdCallAlloc1@@YGPAXK@Z

This custom memory allocation function takes one parameter, a size and returns a pointer to the allocated memory.

• Click the Add… button to create a custom hook definition. The custom hook definition dialog is displayed.
• Type the DLL name testCustomDLL.dll into the DLL Name field.
• Type the function name ?customStdCallAlloc1@@YGPAXK@Z into the Function Name field.
• Leave the function ordinal value at -1, as the function is not being hooked by ordinal.
• Select the STDCALL call radio box.
• The function has one parameter, type 1 in the Number of Parameters field.
• The function is a memory allocation function, choose Alloc in the Function Purpose field.
• Enable or disable the custom hook as appropriate using the Enabled check box.
• If you want Memory Validator to parse the source code of the calling function to determine a datatype, leave the Datatype field empty, otherwise type the name of the datatype into the Datatype field.
• Click the Add button to add a return value. Leave the values unchanged.
• Click the Add button to add a parameter. Click in the Type field and choose Size.

The dialog should look like the image below. Click OK to accept the function definition.

Adding custom reallocation hook

Our example memory allocation function is

    extern void __stdcall customStdCallReAlloc2(DWORD size,    // input param
                                                void     *ptr1,   // input param
                                                void     **ptr2); // output param

and has a mangled name of

    ?customStdCallReAlloc2@@YGXKPAXPAPAX@Z

This custom memory allocation function takes one parameter, a size and returns a pointer to the allocated memory.

• Click the Add… button to create a custom hook definition. The custom hook definition dialog is displayed.
• Type the DLL name testCustomDLL.dll into the DLL Name field.
• Type the function name ?customStdCallReAlloc2@@YGXKPAXPAPAX@Z into the Function Name field.
• Leave the function ordinal value at -1, as the function is not being hooked by ordinal.
• Select the STDCALL call radio box.
• The function has three parameters, type 3 in the Number of Parameters field.
• The function is a memory reallocation function, choose Realloc in the Function Purpose field.
• Enable or disable the custom hook as appropriate using the Enabled check box.
• If you want Memory Validator to parse the source code of the calling function to determine a datatype, leave the Datatype field empty, otherwise type the name of the datatype into the Datatype field.
• Click the Add button to add a parameter. Click in the Parameter field and choose 0. Click in the Type field and choose Size.
• Click the Add button to add a parameter. Click in the Parameter field and choose 1. Click in the Type field and choose Pointer.
• Click the Add button to add a parameter. Click in the Parameter field and choose 2. Click in the Usage field and choose Out. Click in the Type field and choose Pointer to Pointer.

The dialog should look like the image below. Click OK to accept the function definition.

Adding custom free hook

Our example memory allocation function is

    extern void  __stdcall customStdCallFree1(void *ptr);   // input param

and has a mangled name of

    ?customStdCallFree1@@YGXPAX@Z

This custom memory allocation function takes one parameter, a pointer. The function does not return any value.

• Click the Add… button to create a custom hook definition. The custom hook definition dialog is displayed.
• Type the DLL name testCustomDLL.dll into the DLL Name field.
• Type the function name ?customStdCallFree1@@YGXPAX@Z into the Function Name field.
• Leave the function ordinal value at -1, as the function is not being hooked by ordinal.
• Select the STDCALL call radio box.
• The function has one parameter, type 1 in the Number of Parameters field.
• The function is a memory allocation function, choose Free in the Function Purpose field.
• Enable or disable the custom hook as appropriate using the Enabled check box.
• If you want Memory Validator to parse the source code of the calling function to determine a datatype, leave the Datatype field empty, otherwise type the name of the datatype into the Datatype field.
• Click the Add button to add a parameter. Click in the Type field and choose Pointer.

The dialog should look like the image below. Click OK to accept the function definition.

The group of three hooks are displayed in custom hook tab of the settings dialog. You can define as many custom hooks as you have need to – there is no limit to the number of hooks that can be defined. Each custom hook can have no more than 64 parameters.

 

 

How to tell if a function is __cdecl or __stdcall

This section refers to Microsoft calling conventions for Visual Studio C/C++. If you are using another language or compiler, you should choose the appropriate calling convention based on your knowledge of your compiler.

You should consult your function prototypes to determine the calling convention. Typically if you have using extern “C” the calling convention is __cdecl. Most Win32 functions are __stdcall. C++ functions are “thiscall”. “thiscall” is not a compiler keyword, but is an implied calling convention used for C++ classes, with the “this” parameter passed in the ECX register. A fourth convention __fastcall also exists in which the first two arguments are passed in the ECX and EDX registers.

Sometimes you do not have the appropriate function prototypes to determine if a function calling convention is __cdecl or __stdcall. When this is the case, you need to determine the calling convention by other means.

If the function name in the Export Address Table is a decorated C++ function name and the name has a part that reads “@@YA” or “@@YG” the A means __cdecl, the G means __stdcall.

An alternative method is to inspect the function call in the debugger. Start your application in the debugger. Place a breakpoint at the location in your where the function is called. When the debugger stops at the breakpoint, right click in the source code window and choose “Go To Disassembly” on the context menu.

The source code view will display the assembly language for the function.

CDECL Function

Note that after the call instruction there is an add esp, value instruction to manipulate the stack pointer. The size of value will depend on the number of arguments passed to the function.

STDCALL Function

Note that after the call instruction there is no instruction to manipulate the stack pointer.

Please note the above example is for a debug build – release builds may choose to correct the stack alignment in a different manner, depending on the optimisations in force during a release build.