The Custom Hooks tab allows you to specify hooks for functions that Bug Validator does not know about. This allows you to monitor APIs in 3rd party products and in APIs that are released after Bug Validator was released.

 

This a very advanced topic, requiring you to know the parameter list and return type of each function to be hooked and the calling convention for each function. Failure to specify the information correctly may crash your application. If in doubt do not attempt to use this feature of Bug Validator. This topic is also discussed in the Bug Validator tutorial, which can be found on the Help menu.

 

 

 

 

The picture above shows some custom hook definitions in the testCustomDLL.dll DLL. Some of the definitions are for functions with __cdecl calling conventions and some are for functions with the __stdcall calling convention. To edit an existing definition, double click the entry you want to edit, or click the Edit... button.

 

•Add add a new custom hook. The custom hook dialog is displayed.

 

•Edit... edit the selected custom hook. The custom hook dialog is displayed.

 

•Remove delete the custom hook that is selected on the display. Alternatively you can press the Delete key on the keyboard to delete the currently selected entry.

 

•Remove All delete all custom hook definitions.

 

•Enable All enable all custom hook definitions.

 

•Disable All disable all custom hook definitions.

 

 

Typically, after setting the custom hook definitions, you will not want these reset when you choose to reset the other settings to the defaults by pressing the Reset button. However if you do wish to reset the custom hooks at this point you can select the Allow the Reset button to reset the custom hook definitions check box.

 

 

Custom Hook Dialog

 

The custom hook dialog allows you to edit the definition of a custom hook.

 

A custom hook dialog for a function using the __cdecl calling convention is shown below. The function prototype for this function is:

 

 extern "C" void *customAlloc1(int        size);                // input param

 

The function has been specified using the extern "C" specifier so that the function name has no C++ name mangling decoration. The function is the called customAlloc1 and is in the DLL testCustomDLL.dll, it uses the __cdecl calling convention, takes one input parameter. The custom hook is marked as an allocator by the Alloc definition. For datatype purposes it has been specified as returning the void *datatype. The custom hook will monitor the return parameter, marked as a pointer and the single input parameter, marked as a size specifier.

 

 

 

DLL Name

Type the name of the DLL without a file path in the DLL Name field. The name is case insensitive.

 

Function Name

Type the name of the function in the Function Name field. The name should appear exactly as shown in the Export Address Table of the DLL (or the Import Address Table of any DLL that uses the function).

 

Function Ordinal

If the function is imported by ordinal (for example MAPIAllocateBuffer@8 in MAPI32.DLL), type the ordinal (as a decimal integer) in the Function Ordinal field. If the function is not imported by ordinal, type -1.

 

For the example MAPIAllocateBuffer@8 in MAPI32.DLL, the ordinal is 13. The ordinal can be found in the Export Address Table of the DLL.

 

Calling convention

Select the radio box that corresponds to the calling convention of the function. If you don't know the calling convention, DO NOT GUESS. Using the wrong calling convention will crash your application.

 

Number of Parameters

Type the number of parameters the function takes. This is the number of parameters the function has, not the number of parameters that you want to monitor. If you don't know the number of function parameters, DO NOT GUESS. Using the wrong number of function parameters will crash your application when used with the stdcall calling convention.

 

Enabled

Select the check box to make the custom hook enabled. Deselect the check box to disable the custom hook. This setting only takes effect each time your application starts - changing this in the middle of a session has no effect.

 

Return type

The parameters section allows you to define the return value to be monitored.

 

Parameters

The parameters section allows you to define the parameters to be monitored.

 

Add

Click Add to add a parameter definition.

 

Remove

Click Remove to remove the selected parameter.

 

Remove All

Click Remove All to remove all parameters.

 

The return types and each parameter definition can be edited by clicking on the field in the grid. A combo box will be displayed with appropriate choices. The various columns are:

 

Name

This column is the name of the parameter.

 

Type

This column indicates what datatype the parameter is. The following values are valid.

 

void

char

wchar_t

TCHAR

int

short

long

WORD

DWORD

float;

double

         

void *

char *

wchar_t *

TCHAR *

int *

short *

long *

WORD *

DWORD *

float *

double *

 

 

Reset All - Resets all global settings, not just those on the current page.

 

 

Reset - Resets the settings on the current page.