DRAFT: Take more than your usual care.

The User-Mode WMI_LOGGER_CONTEXT

Right from its introduction for Windows 2000, before it even had its current name, Event Tracing for Windows (ETW) has parallel implementations in kernel and user modes. The practical difference is not just with where the code is but with where the trace buffers are managed. One aim of ETW is that events are written quickly into trace buffers which are then serviced independently from a logger thread, e.g., to flush full buffers to an Event Trace Log (ETL) file. An event provider that executes as a user-mode process can choose the user-mode ETW implementation so that the trace buffers are managed in user mode and writing an event (to a trace buffer) avoids the delay of going to and from kernel mode. The user-mode WMI_LOGGER_CONTEXT structure is what NTDLL—and before it, ADVAPI32—keeps for managing an event logger, known more formally as an event tracing session, that has user-mode trace buffers.

Documentation Status

Microsoft seems never to have disclosed even a name for this structure. It is here named WMI_LOGGER_CONTEXT for its correspondence with a kernel-mode structure for which Microsoft’s names (not only of the structure but also of its members) are known from public symbol files, starting with Windows XP SP3 and Windows Server 2003 SP2. Work backwards through the kernel binaries to the kernel-mode structure’s origin in Windows 2000, and it is abundantly plain that whatever the user-mode structure that’s described in this article has ever been named, it and the kernel-mode WMI_LOGGER_CONTEXT started with a shared definition: what members they have in common are laid out in the same order. Though the layouts have since diverged, such that they seem unlikely to be differentiated just by preprocessor directives for conditional compilation, they surely still are developed each with one eye on the other.

The wonder is that Microsoft’s names and types for this structure haven’t slipped out long before. After all, its kernel-mode correspondent seems to be no more secret than many another internal detail, if only to be disclosed as an aid to kernel-mode debugging. Perhaps I’ve just missed the publication or not thought what obvious names to search for.

Or not. User-mode event tracing has much about it that Microsoft demonstrably doesn’t document. This applies especially to functionality that is implemented for user-mode loggers but not (yet) for tracing through the kernel—and happens often enough to suggest a pattern of meaning very much not to document it. Consider, for instance, that Windows 8 and higher provide for user-mode loggers to compress what they flush to ETL files, but this useful trick isn't implemented for kernel-mode tracing until later releases of Windows 10. Any programmer or administrator who has ever worried how big their user-mode tracing session’s ETL files may get will plausibly think that the EVENT_TRACE_COMPRESSED_MODE flag for the logging mode might be a helpful thing to know. Microsoft surely agrees, else why does the Start-EtwTraceSession command in PowerShell have a -Compress switch? Yet, if only according to Google, today, 18th December 2018, the whole of Microsoft’s website has just one page that mentions this flag that programmers would need to know for starting an event tracing session that compresses its own ETL file—and this page is not programmer documentation but PowerShell user documentation. For programmers, EVENT_TRACE_COMPRESSED_MODE didn’t even get defined in EVNTRACE.H until the Software Development Kit (SDK) for the 1607 edition of Windows 10 and it still isn’t listed among the Logging Mode Constants. Such lack of disclosure is not credibly by accident or oversight—except, perhaps, that one page slipped through.

Layout

Curiously, the user-mode WMI_LOGGER_CONTEXT has been far more stable than its kernel-mode counterpart. It has grown, of course, and members have come and gone, but there has been only one large-scale rearrangement (for version 6.1) and the move from ADVAPI32 to NTDLL in version 5.2 didn’t change the structure at all. The following changes of size are known:

Versions Size (x86) Size (x64)
5.0 0xD0  
5.1 to 5.2 0xD8 0x0120
6.0 0xF8 0x0150
6.1 0x0120 0x01A0
6.2 0x0160 0x0210
6.3 to 10.0 0x0170 0x0220

These sizes, and the offsets, types and names in the detailed layout below, come from inspection of the binaries for ADVAPI32 (in its versions 5.0 and 5.1) and NTDLL (in its versions 5.2 and higher), and comparison both with similar code in contemporaneous versions of the kernel and with the offsets, types and names that are known for the kernel-mode structure from public symbol files for the kernel. Where correspondence seems close, it seems reasonable to assume that types and names are the same for both the kernel-mode and user-mode structures. Such analysis is inevitably inexact and prone to oversight. Editorial decisions for cases where correspondence is not so close are explained after the table.

Offset (x86) Offset (x64) Definition Versions Remarks
0x00 (5.0 to 6.0) 0x00 (5.2 to 6.0)
LARGE_INTEGER StartTime;
5.0 to 6.0  
0x00 0x00
ETW_REF_CLOCK ReferenceTime;
6.1 and higher  
0x10 0x10
ULONG ClockType;
6.1 and higher previously at 0x94 and 0xD4
0x08 (5.0 to 6.0) 0x08 (5.2 to 6.0)
HANDLE LogFileHandle;
5.0 to 6.0 next at 0x5C and 0x80
0x0C (5.0 to 6.0) 0x10 (5.2 to 6.0) unaccounted four or eight bytes 5.0 to 6.0  
0x10 (5.0 to 6.0);
0x14
0x18 (5.2 to 6.0);
0x14
ULONG LoggerId;
5.0 and higher  
0x14 (5.1 to 6.0) 0x1C (5.2 to 6.0)
ULONG LocalSequence;
5.1 to 6.0 next at 0x0108 and 0x0180
0x14 (5.0);
0x18 (5.1 to 6.0)
0x20 (5.2 to 6.0)
HANDLE LoggerSemaphore;
5.0 to 6.0  
0x18 (5.0);
0x1C (5.1 to 6.0);
0x18
0x28 (5.2 to 6.0);
0x18
HANDLE LoggerThreadId;
5.0 and higher  
0x1C (5.0);
0x20 (5.1 to 5.2);
not in 6.0;
0x1C
0x30 (5.2);
not in 6.0;
0x20
HANDLE LoggerThread;
5.0 to 5.2;
not in 6.0;
6.1 and higher
 
0x20 (5.0);
0x24 (5.1 to 5.2);
0x20 (6.0)
0x38 (5.2);
0x30 (6.0)
HANDLE LoggerEvent;
5.0 to 6.0 next at 0x58 and 0x78
0x24 (5.0);
0x28 (5.1 to 5.2);
0x24 (6.0);
0x20
0x40 (5.2);
0x38 (6.0);
0x28
DWORD LoggerStatus;
5.0 and higher  
0x24 0x2C
ULONG EventMarker [1];
6.2 and higher  
0x28 0x30
ULONG ErrorMarker;
6.2 and higher  
0x2C 0x34
ULONG SizeMask;
6.2 and higher  
0x24 (6.1);
0x30
0x2C
GUID InstanceId;
6.1 and higher previously at 0xC4 and 0x0108
0x34 (6.1);
0x40
0x3C
ULONG InstanceCount;
6.1 and higher  
0x38 (6.1);
0x44
0x40 (6.1);
0x50
unknown CONDITION_VARIABLE 6.1 and higher  
0x3C (6.1);
0x48
0x48 (6.1);
0x58
unknown CRITICAL_SECTION 6.1 and higher previously at 0xD8 and 0x0120
0x54 (6.1);
0x60
0x70 (6.1);
0x80
HANDLE FlushEvent;
6.1 and higher  
0x58 (6.1);
0x64
0x78 (6.1);
0x88
HANDLE LoggerEvent;
6.1 and higher previously at 0x20 and 0x30
0x5C (6.1);
0x68
0x80
HANDLE LogFileHandle;
6.1 and higher previously at 0x08
0x28 (5.0);
0x2C (5.1 to 5.2);
0x28 (6.0)
0x44 (5.2);
0x3C (6.0)
LONG BuffersAvailable;
5.0 to 6.0 next at 0x98 and 0xD8
0x2C (5.0);
0x30 (5.1 to 5.2);
0x2C (6.0)
0x48 (5.2);
0x40 (6.0)
ULONG NumberOfProcessors;
5.0 to 6.0 next at 0x7C and 0xBC
0x30 (5.0);
0x34 (5.1 to 5.2);
0x30 (6.0)
0x4C (5.2);
0x44 (6.0)
unknown 32-bit page-aligned buffer size 5.0 to 6.0 next at 0x88 and
0x34 (5.0);
0x38 (5.1 to 5.2);
0x34 (6.0)
0x50 (5.2);
0x48 (6.0)
LIST_ENTRY FreeList;
5.0 to 6.0  
0x3C (5.0);
0x40 (5.1 to 5.2);
0x3C (6.0)
0x60 (5.2);
0x58 (6.0)
LIST_ENTRY GlobalList;
5.0 to 6.0 next at 0x9C and 0xE0
0x44 (5.0);
0x48 (5.1 to 5.2);
0x44 (6.0)
0x70 (5.2);
0x68 (6.0)
unknown pointer 5.0 to 6.0  
0x48 (5.0);
0x4C (5.1 to 5.2);
0x48 (6.0)
0x78 (5.2);
0x70 (6.0)
WMI_BUFFER_HEADER **ProcessorBuffers;
5.0 to 6.0  
0x4C (5.0);
0x50 (5.1);
0x4C (6.0);
0x60 (6.1);
0x6C
0x80 (5.2);
0x78 (6.0);
0x88 (6.1);
0x98
UNICODE_STRING LoggerName;
5.0 and higher  
0x54 (5.0);
0x58 (5.1);
0x54 (6.0);
0x68 (6.1);
0x74
0x90 (5.2);
0x88 (6.0);
0x98
UNICODE_STRING LogFileName;
5.0 and higher  
0x70 (6.1);
0x7C
0xA8
UNICODE_STRING LogFilePattern;
6.1 and higher  
0x78 (6.1);
0x84
0xB8
LONG FileCounter;
6.1 and higher  
0x7C (6.1);
0x88
0xBC
ULONG NumberOfProcessors;
6.1 and higher previously at 0x2C and 0x40
0x80 (6.1);
0x8C
0xC0 (6.1);
0xD0
ULONG BufferSize;
6.1 and higher previously at 0xA0 and 0xE0
0x84 (6.1);
0x90
 
ULONG MaximumEventSize;
6.1 and higher  
0x88 (6.1);
0x94
  unknown 32-bit page-aligned buffer size 6.1 and higher previously at 0x30 and 0x44
0x8C (6.1);
0x98
0xCC (6.1);
0xDC
ULONG MaximumBuffers;
6.1 and higher previously at 0xA8 and 0xE8
0x90 (6.1);
0x9C
0xD0 (6.1);
0xE0
ULONG MinimumBuffers;
6.1 and higher previously at 0xAC and 0xEC
0x94 (6.1);
0xA0
0xD4 (6.1);
0xE4
ULONG NumberOfBuffers;
6.1 and higher previously at 0xA4 and 0xE4
0x98 (6.1);
0xA4
0xD8 (6.1);
0xE8
LONG BuffersAvailable;
6.1 and higher previously at 0x28 and 0x3C
0x9C (6.1);
0xA8
0xE0 (6.1);
0xF0
LIST_ENTRY GlobalList;
6.1 and higher previously at 0x3C and 0x58
0xA4 (6.1);
0xB0
0xF0 (6.1);
0x0100
ETW_BUFFER_QUEUE BufferQueue;
6.1 and higher  
0xB0 (6.1);
0xBC
0x0108 (6.1);
0x0118
unknown ETW_BUFFER_QUEUE 6.1 and higher  
0xBC (6.1);
0xC8
0x0120 (6.1);
0x0130
ETW_BUFFER_QUEUE OverflowQueue;
6.1 and higher  
0x5C (5.0);
0x60 (5.1 to 5.2);
0x5C (6.0)
0xA0 (5.2);
0x98 (6.0)
LONG CollectionOn;
5.0 to 6.0 next at 0xD0 and 0x0140
0x60 (5.0);
0x64 (5.1 to 5.2)
0xA4 (5.2) unaccounted eight bytes 5.0 to 5.2  
0x60 (6.0) 0x9C (6.0)
ULONG RequestFlag;
6.0 only next at 0xD4 and 0x0144
0x68 (5.0);
0x6C (5.1 to 5.2);
0x64 (6.0);
0xC8 (6.1);
0xD4
0xAC (5.2);
0xA0 (6.0);
0x0138 (6.1);
0x0148
ULONG MaximumFileSize;
5.0 and higher  
0x6C (5.0);
0x70 (5.1 to 5.2);
0x68 (6.0);
0xCC (6.1);
0xD8
0xB0 (5.2);
0xA4 (6.0);
0x013C (6.1);
0x014C
ULONG LoggerMode; 
5.0 and higher  
0xD0 (6.1);
0xDC
0x0140
LONG CollectionOn;
6.1 and higher previously at 0x5C and 0x98
0xD4 (6.1);
0xE0
0x0144
union {
    ULONG RequestFlag;
    struct {
        /*  changing bit fields  */
    };
};
6.1 and higher previously at 0x60 and 0x9C
0x70 (5.0);
0x74 (5.1 to 5.2);
0x6C (6.0);
0xD8 (6.1);
0xE4
0xB4 (5.2);
0xA8 (6.0);
0x0148
ULONG LastFlushedBuffer;
5.0 and higher  
0x78 (5.0 to 5.2);
0x70 (6.0);
0xE0 (6.1);
0xE8
0xB8 (5.2);
0xB0 (6.0);
0x0150 (6.1);
0x0160
LARGE_INTEGER FlushTimer;
5.0 and higher  
0x80 (5.0 to 5.2);
0x78 (6.0);
0xE8
0xC0 (5.2);
0xB8 (6.0);
0x0158
LARGE_INTEGER FirstBufferOffset;
5.0 and higher  
0x88 (5.0 to 5.2);
0x80 (6.0);
0xF0 (6.1);
0xF8
0xC8 (5.2);
0xC0 (6.0);
0x0160
LARGE_INTEGER ByteOffset;
5.0 and higher  
0x90 (5.0 to 5.2);
0x88 (6.0)
0xD0 (5.2);
0xC8 (6.0)
LARGE_INTEGER BufferAgeLimit;
5.0 to 6.0  
0x98 (5.1 to 5.2);
0x90 (6.0)
0xD8 (5.2);
0xD0 (6.0)
unaccounted four bytes 5.1 to 6.0  
0x9C (5.1 to 5.2):
0x94 (6.0)
0xDC (5.2);
0xD4 (6.0)
ULONG ClockType;
5.1 to 6.0 next at 0x10
0x98 (6.0) 0xD8 (6.0)
LARGE_INTEGER ReferenceTimeStamp;
6.0 only  
0x98 (5.0);
0xA0 (5.1 to 6.0)
0xE0 (5.2 to 6.0)
ULONG BufferSize;
5.0 to 6.0 next at 0x80 and 0xC0
0x9C (5.0);
0xA4 (5.1 to 6.0)
0xE4 (5.2 to 6.0)
ULONG NumberOfBuffers;
5.0 to 6.0 next at 0x94 and 0xD4
0xA0 (5.0);
0xA8 (5.1 to 6.0)
0xE8 (5.2 to 6.0)
ULONG MaximumBuffers;
5.0 to 6.0 next at 0x8C and 0xCC
0xA4 (5.0);
0xAC (5.1 to 6.0)
0xEC (5.2 to 6.0)
ULONG MinimumBuffers;
5.0 to 6.0 next at 0x90 and 0xD0
0x0100   unaccounted eight bytes 6.2  
0x0100 0x0178
ULONG FlushThreshold;
6.3 and higher  
         
0xA8 (5.0);
0xB0 (5.1 to 6.0);
0xF8 (6.1);
0x0108 (6.2);
0x0110
0xF0 (5.2 to 6.0);
0x0168 (6.1);
0x0180 (6.2);
0x0188
ULONG EventsLost;
5.0 and higher  
0xFC (6.1);
0x010C (6.2);
0x0114
0x016C (6.1);
0x0184 (6.2);
0x018C
ULONG LogBuffersLost;
6.1 and higher previously at 0xB8 and 0xF8
0xAC (5.0);
0xB4 (5.1 to 6.0);
0x0100 (6.1);
0x0110 (6.2);
0x0118
0xF4 (5.2 to 6.0);
0x0170 (6.1);
0x0188 (6.2);
0x0190
ULONG BuffersWritten;
5.0 and higher  
0xB0 (5.0);
0xB8 (5.1 to 6.0)
0xF8 (5.2 to 6.0)
ULONG LogBuffersLost;
5.0 to 6.0 next at 0xFC and 0x016C
0xB4 (5.0);
0xBC (5.1 to 6.0)
0xFC (5.2 to 6.0)
ULONG RealTimeBuffersLost;
5.0 to 6.0  
0xC0 (5.1 to 6.0);
0x0104 (6.1);
0x0114 (6.2);
0x011C
0x0100 (5.2 to 6.0);
0x0178
LONG *SequencePtr;
5.1 and higher  
0x0108 (6.1);
0x0118 (6.2);
0x0120
0x0180
ULONG LocalSequence;
6.1 and higher previously at 0x14 and 0x1C
0xB8 (5.0);
0xC4 (5.1 to 6.0)
0x0108 (5.2 to 6.0)
GUID InstanceId;
5.0 to 6.0 next at 0x24 and 0x2C
0xD4 (6.0) 0x0118 (6.0) unknown dword 6.0 only  
0xD8 (6.0) 0x0120 (6.0) unknown CRITICAL_SECTION 6.0 only next at 0x3C and 0x48
0x0110 (6.1);
0x0120 (6.2);
0x0128
0x0188
LONGLONG BufferSequenceNumber;
6.1 and higher  
0xC8 (5.0);
0xD4 (5.1 to 5.2);
0xF0 (6.0);
0x0118 (6.1);
0x0128 (6.2);
0x0130
0x0118 (5.2);
0x0148 (6.0);
0x0190
unknown pointer to trace buffers 5.0 and higher last member in 5.0 to 6.0

There is something to write here!

Offset (x86) Offset (x64) Definition Versions
0x012C (6.2);
0x0134
0x01B0 (6.2);
0x01B8
PVOID CompressionWorkspace;
6.2 and higher
0x0130 (6.2);
0x0138
0x01B8 (6.2);
0x01C0
buffer for compressed data 6.2 and higher
0x0134 (6.2);
0x013C
0x01C0 (6.2);
0x01C8
32-bit size of compression buffer 6.2 and higher
0x0138 (6.2);
0x0140
0x01C4 (6.2);
0x01CC
32-bit size of trace buffers in compression buffer 6.2 and higher
0x0144 0x01D0 32-bit count of trace buffers in compression buffer 6.3 and higher
0x0148 0x01D4 32-bit size of partial trace buffer in compression buffer 6.3 and higher

As noted in the introduction, user-mode loggers can compress event data in version 6.2 and higher but there is no kernel-mode correspondent until the 1607 release of Windows 10. The implementations are very different. The only known name that seems safe to take as common to both implementations is that of the buffer that’s used for the algorithm’s workspace.

A user-mode logger compresses successive trace buffers ever deeper into a compression buffer that’s the size of two trace buffers. The first is flushed whenever it fills. Any overflow is then moved down to become a partial trace buffer at the start of the compression buffer. See that any one flush from the compression buffer is of one trace buffer’s worth of data that has been compressed from potentially many trace buffers. Version 6.2 counts them altogether as one buffer written or lost. Version 6.3 tracks their number. If flushing fails, version 6.2 does not reset its compression buffer. In later versions, all whole trace buffers that were in the compression buffer are explicitly lost. If the compression buffer began with the overflow data of a partly flushed trace buffer, then this overflow is retained in the compression buffer so that any subsequent flush that does succeed will complete the partial trace buffer that is already in the ETL file.

Offset (x86) Offset (x64) Definition Versions
0x013C (6.2);
0x014C
0x01C8 (6.2);
0x01D8
LIST_ENTRY ProviderBinaryList;
6.2 and higher
0x0144 (6.2);
0x0154
0x01D8 (6.2);
0x01E8
LIST_ENTRY WinRtProviderBinaryList;
6.2 and higher

It seems strange now, but not until version 6.2 do ETL files record which providers are enabled for the session.

Offset (x86) Offset (x64) Definition Versions
0x014C (6.2);
0x015C
0x01F8 unknown dword 6.2 and higher
0x0150 (6.2);
0x0160
0x0200 unknown pointer to array of pointers 6.2 and higher
0x0154 (6.2);
0x0164
0x0208 unknown pointer to array of structures 6.2 and higher
0x0158 (6.2);
0x0168
0x0200 (6.2);
0x0210
unknown pointer to processor stream index map 6.2 and higher

There is something to write here!

Offset (x86) Offset (x64) Definition Versions Remarks
0x011C (6.1);
0x015C (6.2);
0x016C
0x0198 (6.1);
0x0208 (6.2);
0x0218
WMI_BUFFER_HEADER *ProcessorBuffers [ANYSIZE_ARRAY];
6.1 and higher last member in 6.1 and higher

Where the kernel-mode WMI_LOGGER_CONTEXT has objects such as semaphores and events, and pointers to objects such as threads, a user-mode structure can only have handles. Although the applicable members must change type, they needn’t change names and it is assumed above that they don’t.

Each logger creates its own thread for flushing trace buffers. All versions keep the thread ID and disclose it, when queried, by copying it to the LoggerThreadId member of a WMI_LOGGER_INFORMATION. It is here proposed that the thread ID as kept in the logger context likely has the same name (and type). The early versions, implemented in ADVAPI32, also keep a handle to the logger thread. This stops with version 5.2, but it is assumed above that the LoggerThread remained defined (no reuse of its space being known). Version 6.0 seems then to have removed it, only to have version 6.1 go back to keeping the handle.

The unknown CONDITION_VARIABLE supports EVENT_TRACE_BLOCKING_MODE. If tracing an event would exhaust its processor’s current trace buffer but all other trace buffers are in use (including because they are yet to be flushed) and no more can be created, then the event would ordinarily be lost. Waiting for a trace buffer to become available isn’t much of an option. After all, much of the point to tracing an event is that it is done without disturbing whatever activity is being traced. For some loggers, however, the intended activity may be relatively infrequent and take time anyway, such that time spent waiting for a buffer to trace an event to is less a penalty than would be the event’s loss. Blocking mode allows the wait. It’s another ETW feature that’s particular to user-mode tracing sessions (and is not documented).

See that in all versions, a user-mode logger reserves address space sufficient for whatever it ends up adopting as MaximumBuffers. It immediately commits memory for whatever it ends up adopting as MinimumBuffers, and thereafter commits more only when more buffers actually are needed for whatever flow of events are received. For this purpose of reserving and committing, each trace buffer is a whole number of pages, rounded up from BufferSize bytes. Neither the page-aligned buffer size nor the address of the reservation has any correspondent in the kernel-mode structure.

Versions 6.1 and higher end the structure with a variable-size array of pointers, one per processor, each to the trace buffer that is currently in use for that processor. This too has no correspondence with the kernel-mode structure. It does, however, supersede what had been a pointer to just such an array, and in such a way that keeping the name would require no change to the source code. The pointer’s name, ProcessorBuffers, is known from correspondence with the kernel-mode structure.