The Custom Hooks tab allows you to specify hooks for functions that Memory Validator does not initially know about.
For example, custom hooks might let you monitor APIs in third party products or in APIs that are released after Memory Validator was released.
This a very advanced topic.
Read on, or click on a setting in the picture below to find out more
Using custom hooks
Before adding custom hooks, you need to know the parameter list and return type of each function to be hooked and their calling conventions
.
•Add... 
displays the custom hook dialog (below)
For other options (edit,remove, and enable), see modifying existing custom hooks further below.
This topic is also discussed in the Memory Validator tutorial, which can be found on the Help menu.
Custom hook dialog
The custom hook dialog allows you to set up or edit the definition of a custom hook, including its parameters.
Take care! Failure to specify the information correctly may crash your application. If in doubt do not attempt to use this feature of Memory Validator.
At the end of this topic are some examples of how this dialog might be filled out for a few different functions.
•DLL Name Browse to navigate to a DLL Open to enter the DLL name into the dialog
Alternatively you can type the full path, or a relative path to the DLL into the DLL Name field.
•Function Name choose the name of the exported function from the drop-down list
The list is automatically populated with the exported functions after choosing a DLL name above
The function name must be exported from the DLL you specified in order to appear in the list. If the function is not exported it cannot be hooked.
•Function Ordinal type the ordinal (decimal integer) if the function is imported by ordinal, otherwise enter -1 to ignore it
For example MAPIAllocateBuffer@8 in MAPI32.DLL has the ordinal 13 as found in the Export Address Table of the DLL
Finding the ordinal of an exported function in a DLL One way to find the ordinal of a function exported from a DLL is to use the dumpbin.exe utility provided with Microsoft Visual Studio. e.g. dumpbin filename.dll /EXPORTS
In Visual Studio 9.0, the dumpbin.exe utility is found in VC\bin and is dependent on link.exe in the same directory and mspdb80.dll in the Common7\IDE directory.
•Calling convention select the option that corresponds to the calling convention of the function
Do not guess this, if you don't know! Using the wrong calling convention will crash your application
See also, the history of calling conventions
•Number of Parameters enter the number of parameters the function takes
This is the number of parameters the function has, not the number of parameters that you want to monitor (below).
Again - do not guess this! Using the wrong number of function parameters will crash your application when used with the stdcall calling convention.
•Function Purpose set the function purpose - Alloc, AllocMore, Realloc, or Free
•Enabled tick to enable hooking of the function
Only takes effect each time your application starts, so changing this in the middle of a session has no effect.
•Datatype enter the type of all allocations via this hook, or leave empty and Memory Validator will parse the source code to try and obtain the datatype
Do not include parentheses in the datatype. [] and <> are acceptable though.
Custom hook function parameters
The parameters section of the dialog allows you to define which parameters and return value are monitored for the function.
You should specify all the other settings in the custom hook definition (see section above) prior to defining the parameters. This will ensure that the right menu choices are made available while setting up the parameters.
You don't need to monitor all the input parameters, or the return value - just monitor what you need.
Generally for each function purpose (that you will have defined above), different parameter values are usually required:
For example:
•Alloc an input size, and an output pointer or return value pointer
•AllocMore an input size and an input pointer
•Realloc an input size, an input pointer, and an output pointer or return value pointer
•Free an input pointer
In some cases, an input parameter may also serve as an output.
If you want to monitor additional parameters, we recommend that you set their type to Miscellaneous or Pointer to Miscellaneous as appropriate.
•Add adds a row to the parameters table, firstly a return value, then parameters in order 0,1,2...
•Remove removes selected parameter definitions from the list
•Remove All clears the list of parameters
Defining custom hook function parameters
After adding a parameter definition, values can be edited by double clicking on the value in the table. A drop-down list will be displayed with appropriate choices in each column:
•Parameter choose the parameter index in the range 0 to N-1 where N is the number of parameters you specified to the function
You can only choose an index which is not already used by another parameter
You can't change the entry for a "return value"
•Usage set how the parameter is used:
In is input only and Out is output only, while In/Out serves as both
Return value, this and any output parameters are usually pointers to something.
•Type choose what the parameter type is:
All Pointer to... options are parameters that need dereferencing to read the values.
Memory Validator x64 also supports the following values:
Size (QWORD) a size
Pointer to size (QWORD) a pointer to a size
Miscellaneous (QWORD) a miscellaneous value
Pointer to Miscellaneous (QWORD) a pointer to a miscellaneous value
Note that when specifying parameters as Pointer to ..., it's OK if that parameter is occasionally NULL - this will be identified and the pointer will not be dereferenced. Many Microsoft APIs allow NULL values for optional data, for example.
Modifying existing custom hooks
Once you have some custom hooks set up you can edit, remove or en/disable them in the following ways:
•Edit... or double click a hook in the list opens the custom hook dialog to change the hook attributes
The usual controls apply for removing list items:
•Remove removes selected items from the list
•Remove All clears the list
Alternatively, press 
to delete selected items, and 
+ 
to select all items in the list first.
Enable or disable a custom hook by ticking or unticking the yellow boxes or change them all at once
•Enable All ticks all the hooks, enabling them all
•Disable All unticks everything
•Import... Imports a custom hook XML file.
•Export... Exports the custom hooks as an XML file. This can be useful for moving a custom hook definitions from one computer to another without copying the whole settings file.
Resetting custom hooks
Normally, the custom hooks will not be reset when you press the reset button, but you can make that the case if you wish:
•Allow the Reset button to reset the custom hook definitions when ticked, custom hooks will be reset with all other global settings
Reset All - Resets most global settings including those on other pages, but not the settings on this page unless explicitly requested above.
Reset - Resets the settings on the current page.
These examples show how to use the custom hooks dialog
Custom hook dialog - example 1
A custom hook dialog for a function using the __cdecl calling convention is shown below.
The function prototype for customAlloc1 in the DLL testCustomDLL.dll (provided) is:
extern "C" void *customAlloc1(DWORD size); // input param
It has been specified using the extern "C" specifier so that the function name has no C++ name mangling decoration.
It uses the __cdecl calling convention, takes one input parameter and is marked as an allocator by the Alloc definition.
For datatype purposes it can be specified as returning the BYTE datatype.
The custom hook will monitor:
•the return parameter, marked as a pointer
•the single input parameter, marked as a size specifier.
Custom hook dialog - example 2
A custom hook dialog for a function using the __stdcall calling convention is shown below.
The function prototype for ?customStdCallAlloc3@@YGXKPAPAX@Z is in the DLL testCustomDLL.dll is:
extern void __stdcall customStdCallAlloc3(DWORD size, // input param
void **ptr); // output param
It has been compiled as C++, having C++ name mangling decoration.
It uses the __stdcall calling convention, and takes two input parameters.
The function purpose as an allocator is defined by the Alloc setting.
For datatype purposes it can be specified as returning the BYTE <STDCALL> datatype.
The custom hook will monitor:
•the first input parameter, marked as a size specifier
•the second output parameter, marked as a pointer to pointer.
Custom hook dialog - example 3
Similar to example 2, except we have a different function name and its purpose is now as a Realloc function, with a pointer return type.
The function prototype for customStdCallReAlloc1 is:
extern void* __stdcall customStdCallReAlloc1(DWORD size, // input param
void **ptr); // output param
The custom hook will monitor:
•the return parameter as a pointer
•the first input parameter, marked as a size specifier
•the second input parameter, marked as a pointer to pointer.
