More than I ever wanted to know about Win32 calling conventions...

Just spent the last day trying to debug a weird bug - one of those works in Debug builds but crashes in Release builds issues in a C++ MFC application. After tweaking some compiler settings I discovered it wouldn't crash if I disabled optimizations in release mode, but rather than doing that I thought I would dig down to find the root cause.
After much setting of breakpoints and attaching the debugger I found it was crashing in a call to CString::Format() - but only subsequent to a call to a function in 3rd party DLL that was used for encryption. The origins of this DLL were long ago lost in the mists of time so unfortunately there was no documentation.
The function was called by dynamically loading the DLL and calling ::GetProcAddress() to obtain the function pointer, which had the following signature:

typedef long ( __stdcall* LPFNDLLENCRYPT)(const char *, const char *, char *);

__stdcall is used to call most of the Win32 apis. This calling convention can generate smaller code, as the called function performs the stack cleanup code.

__cdecl calling convention relies on the caller to cleanup the stack frame, and since functions are typically called from more than one place in the code, the additional stack cleanup code bloats the code.

On a hunch, I changed the function signature to:

typedef long ( __cdecl* LPFNDLLENCRYPT)(const char *, const char *, char *);

and lo and behold - it fixed the bug!

It seems the DLL was using __cdecl calling convention and expected the caller to cleanup the stack frame, but since the function pointer incorrectly declared it as __stdcall the compiler was not generating the stack cleanup code. This left the stack in a corrupted state which caused the next call to the CString::Format function in the MFC DLL to fail. Presumably the Debug build was more resilient with respect to stack corruption.

Here's a really useful article on calling conventions on CodeProject which explains calling conventions in more detail than you would ever want to know.... unless your debugging weird bugs due to calling conventions!

I love C++!