64 bit porting gotcha #1! x64 Datatype misalignment.
Datatype misalignment, there is a topic so interesting you’d probably prefer to watch paint dry.
But! There are serious consequences for getting it wrong. So perhaps you’d better read about it after all 🙂
The problem that wasted my time
Why am I writing about datatype misalignment? Because its just eaten two days of my time and if what I share with you helps save you from such trouble, all the better.
The problem I was chasing was that three calls to CreateThread() were failing. All calls were failing with ERROR_NOACCESS. They would only fail if called from functions in the Thread Validator x64 profiling DLL injected into the target x64 application. If the same functions were called later in the application (via a Win32 API hook or directly from the target application) the functions would work. That meant that the input parameters were correct.
Lots of head scratching and trying many, many variations of input parameters and asking questions on Stack Overflow and we were stuck. I could only think it was to do with the callstack but I had no idea why. So I started investigating the value of RSP during the various calls. The investigating what happened if I pushed more data onto the stack to affect the stack pointer. After some trial and error I found a combination that worked. Then I experimented with that combination to determine if it was the values being pushed that were important or the actual value of the stack pointer that was important.
At this point I was confused, as I didn’t know about any stack alignment requirements, I only knew about data alignment requirements. I then went searching for appropriate information about stack alignments and found this handy document from Microsoft clarifies that.
What is datatype misalignment?
Datatype alignment is when the data read by the CPU falls on the natural datatype boundary of the datatype. For example, when you read a DWORD and the DWORD is aligned on a 4 byte boundary.
In the following code examples, let us assume that data points to a location aligned on a four byte boundary.
void aligned(BYTE *data)
{
DWORD dw;
dw = *(DWORD *)&bp[4];
}
Datatype misalignment is when the data read by the CPU does not fall on the natural datatype boundary of the datatype. For example, when you read a DWORD and the DWORD is not aligned on a 4 byte boundary.
void misaligned(BYTE *data)
{
DWORD dw;
dw = *(DWORD *)&bp[5];
}
Why should I care about datatype misalignment?
Aligned data reads and data writes happen at the maximum speed the memory subsystem and processor can provide. For example to read an aligned DWORD, one 32 bit data read needs to be performed.
DWORD BYTE [ 0]; BYTE [ 1]; BYTE [ 2]; BYTE [ 3]; DWORD BYTE [ 4]; // ignored BYTE [ 5]; // read BYTE [ 6]; // read BYTE [ 7]; // read DWORD BYTE [ 8]; // read BYTE [ 9]; // ignored BYTE [10]; // ignored BYTE [11]; // ignored DWORD BYTE [12]; BYTE [13]; BYTE [14]; BYTE [15];
Misaligned data reads and data writes do not happen at the maximum speed the memory subsystem and processor can provide. For example to read a misaligned DWORD the processor has to fetch data for the two 32 bit words that the misaligned data straddles.
In the misaligned example shown above, the read happens at offset 5 in the input array. I’ve shown the input array first 16 bytes, marking where each DWORD starts and showing which bytes are read and which are ignored. If we assume the input array is aligned then the DWORD being read has 3 byte2 in the first DWORD and 1 bytes in the second DWORD. The processor has to read both DWORDs, then shuffle the bytes around, discarding the first byte from the first DWORD and discarding the last 3 bytes from the last DWORD, then combining the remaining bytes to form the requested DWORD.
Performance tests on 32 bit x86 processors shown performance drops of between 2x and 3x. That’s quite a hit. On some other architectures, the performance hit can be much worse. This largely depends on if the processor does the rearrangement (as on x86 and x64 processors) or if an operating system exception handler handles it for you (much slower).
I’ve shown the example with DWORDs, because the they are short enough to be easily shown in a diagram whereas 8 byte or larger values would be unwieldy.
The above comments also apply to 8 byte values such as doubles, __int64, DWORD_PTR (on x64), etc.
Clearly, getting your datatype alignments optimized can be very handy in performance terms. Naive porting from 32 bit to 64 bit will not necessarily get you there. You may need to reorganise the order of some data members in your structures. We’ve had to do that with Thread Validator x64.
Not just performance problems either!
In addition to the performance problems mentioned above there is another, more important consideration to be aware of: On x64 Windows operating systems you must have the stackframe correctly aligned. Correct stack alignment on x64 systems means that the stack frame must be aligned on a 16 byte boundary.
Failure to ensure that this is the case will mean that a few Windows API calls will fail with the cryptic error code ERROR_NOACCESS (hex: 0x3E6, decimal: 998). This means “Invalid access to memory location”.
The problem with the error code ERROR_NOACCESS in this case is that the real error code that gets converted into ERROR_NOACCESS is STATUS_DATATYPE_MISALIGNMENT, which tells you a lot more. I spent quite a bit of time digging around until I found the true error code for the bug I was chasing that lead me to write this article.
If you are writing code using a compiler, the compiler will sort the stack alignment details out for you. However if you are writing code in assembler, or writing hooks using dynamically created machine language, you need to be aware of the 16 byte stack alignment requirement.
x64 datatype alignment requirements
Stack:
Correct stack alignment on x64 systems means that the stack frame must be aligned on a 16 byte boundary.
Data:
| Size | Alignment |
|---|---|
| 1 | 1 |
| 2 | 2 |
| 4 | 4 |
| 8 | 8 |
| 10 | 10 |
| 16 | 16 |
Anything larger than 8 bytes is aligned on the next power of 2 boundary.
Conclusion
Correct stack frame alignment is essential to ensure calling functions works reliably.
Correct datatype alignment is essential for maximum speed when accessing data.
Failure to align stack frames correctly could lead to Win32 API calls failing and or program failure or lack of correct behaviour.
Failure to align data correctly will lead to slow speed when accessing data. This could be disasterous for your application, depending upon what it is doing.