Although Exchange 2000 did not include the Performance Optimizer, there were still a variety of registry settings...
and other changes you could make to tune Exchange or change its behavior (see footnote1). Any changes you made to tune or alter the behavior of Exchange 2000 need to be reviewed before implementing Exchange 2003. Indeed, most of them are no longer needed in Exchange 2003, but more importantly some of them can cause problems when used on an Exchange 2003 system. Therefore, before upgrading any Exchange 2000 servers to Exchange 2003, you should review the following information.
(Footnote1: Many of the Exchange 2000 tuning parameters were documented in a Microsoft white paper titled "Microsoft Exchange 2000 Internals: Quick Tuning Guide." If you used this guide to tune your Exchange 2000 server, you likely need to remove some tuning parameters before upgrading to Exchange 2003).
Exchange 2000 included two tuning registry parameters specific to EVSs running in a cluster. Both parameters were designed to prevent a very busy SMTP resource in an EVS from starving other resources in the EVS such as IMAP4 and POP3. The tuning parameters were the SMTP % of threads and Additional threads per processor values, which were represented in the registry as follows.
MaxPercentPoolThreads was used to control the percentage of threads used by the SMTP service, and
AdditionalPoolThreadsPerProc enabled you to control the number of additional threads that could be spawned on a per-processor basis. If you added either of these registry entries to your Exchange 2000 cluster, you should remove them prior to upgrading your EVSs to Exchange 2003.
Exchange Server Directory Service Access cache
As the name suggests, Directory Service Access (DSAccess) is an internal component in Exchange that controls how all Exchange components access Active Directory. The primary function of DSAccess is to keep tabs on various directory-related things. For example, DSAccess discovers the Active Directory topology and detects the state of domain controllers and global catalog servers (up or down). In addition, all directory queries are routed through DSAccess, such as recipient resolution, configuration setting look-ups, and others. As part of its job, DSAccess maintains an in-memory cache of the results of some of these queries so that if the same information is requested twice, it can be retrieved from the DSAccess cache instead of through another LDAP query against Active Directory. The size of the in-memory cache is configurable, in that you can set a maximum size for various cached items.
Many administrators found that on larger Exchange 2000 servers, the out-of-the-box values for the maximum cache size for recipient look-ups and the maximum cache size for configuration look-ups were not always optimized for their servers. On systems with an undersized DSAccess cache, it was common for local message delivery and/or address book ¬resolution to be slow. Once the maximum size of the recipient cache was increased and the size of the configuration cache was decreased, performance would improve. The DSAccess cache tuning parameters for the configuration data cache and the user object cache were represented in the registry as follows.
In Exchange 2000, each cache pool (e.g., the configuration cache, the recipient cache, and so on) was initially set at 25MB in size. To improve performance, the default values for the configuration and recipient caches have been optimized in Exchange 2003. The configuration data cache, which more often than not never needed anywhere near 25MB, now defaults to 5MB. The user object cache, which was often undersized for larger systems, now defaults to 140MB. Therefore, you should remove the MaxMemoryConfig and MaxMemoryUser registry entries prior to upgrading from Exchange 2000 to Exchange 2003.
Extensive storage engine heaps
Like other operating systems, Windows includes a process-wide heap manager that handles memory operations for all processes. Each time a process instantiates, a default heap (called the process heap) is created for that process. Programs or modules loaded in that process can also create additional heaps if needed. Exchange is one program that does this.
In Exchange 2000, each time the Exchange information store was started, the STORE.EXE process got its initial process heap. Then, a module loaded in the information store process—ESE.DLL—allocated four JET heaps for each processor present in the system. Although these heaps were separate pools of memory in the information store process, they were collectively referred to as the ESE multiheap. On a single CPU system, ESE allocated four JET heaps. On a dual CPU system, it allocated eight; on a quad system, it allocated sixteen; and so forth. When Exchange 2000 was installed on systems with four or more CPUs, it was found that the ESE multiheap caused excessive virtual memory consumption, which in turn led to performance problems. Ironically, the problem was worse on systems with a large amount of resources installed (multiple CPUs and multiple gigabytes of memory).
To correct this problem, Microsoft recommended that customers with large Exchange 2000 servers add the following registry entry to their systems.
Value: MPHeap parallelism
The value data setting depended on the number of CPUs present in the system, and if you added or removed CPUs to the system in scale-up or scale-down procedures, you had to manually readjust this setting. Exchange 2003 now automatically calculates the optimum number of heaps to allocate based on the unique specifications of each system. Therefore, you should remove the MPHeap parallelism registry entry from Exchange 2000 servers prior to upgrading to Exchange 2003.
Exchange Server initial memory percentage
In Chapter 7, I wrote about virtual memory and the importance of monitoring it on an Exchange server. It is especially important to watch on Exchange servers for two reasons: (1) the more virtual memory available, the greater the load that can be handled; and (2) performance problems can occur (especially on Exchange clusters) when virtual memory becomes too fragmented. When virtual memory becomes too fragmented, Exchange logs the following event in the Application event log.
Event Type: Error
Event Source: MSExchangeIS
Event Category: Performance
Event ID: 9582
Time: 3:42:38 PM
Description: The virtual memory necessary to run your Exchange server is fragmented in such a way that normal operation may begin to fail. It is highly recommended that you restart all Exchange services to correct this issue.
As this event log entry indicates, restarting all Exchange services was needed in order to correct the problem. If this happened regularly, you ended up restarting Exchange frequently, and both your users and management probably wondered why their email server was so unreliable.
To combat the virtual memory fragmentation problem, Microsoft introduced the following registry entry and recommended value in Exchange 2000 that hard-coded Exchange's initial memory allocation to 10% of the total amount of physical memory in the system.
Value: Initial Memory Percentage
Value Data: 0xa (hex)
By starting out with this initial allocation and then growing from there, virtual memory fragmentation would not occur as often. As I mentioned in Chapter 7, Exchange 2003 includes support for special startup switches in Windows 2000 Advanced Server and Windows Server 2003 that help reduce virtual memory fragmentation, and Microsoft made additional changes to help prevent virtual memory fragmentation in Exchange clusters. As a result, the
Initial Memory Percentage registry value is no longer needed. Moreover, because it does not work on an Exchange 2003 server, you should remove this value from all Exchange 2000 servers prior to upgrading to Exchange 2003.
Exchange Server log buffers
As a database that passes the ACID (Atomic, Consistent, Isolated, and Durable) test for transaction-based activity, Exchange first writes all activity to transaction logs and then commits the transactions to the data¬base file. But before Exchange writes anything to a transaction log, it first holds the information in memory in an area called log buffers. Unlike the settings mentioned so far, which are registry values, the size of the log buffers is controlled by an attribute of the information store object in Active Directory—
Throughout the product life of Exchange 2000, the recommended best practice setting for the
msExchESEParamLogBuffers attribute changed a few times. Out of the box, it was set to 84, which was determined to be too low for heavily used servers, especially clustered back-end servers. When Service Pack 2 for Exchange 2000 was released, Microsoft recommended that customers adjust this value from 84 to 9000. When Service Pack 3 was released, Microsoft recommended that customers readjust the value down from 9000 to 500. Because the out-of-the-box value for
msExchESEParam LogBuffers in Exchange 2003 is 500, you will want to use the ADSI Edit tool to delete any manual tuning entries (even if set to 500) and return the value for this attribute back to
Maximum number of open tables in Exchange Server
Although it typically provided little bang for the buck, another way to combat memory fragmentation and allocation issues in Exchange 2000 was to reduce the maximum number of open tables that can be used by Exchange. Exchange 2000 cached data about folders that were not currently accessed, a behavior that in some cases contributed to virtual memory fragmentation. To reduce the cache's impact on virtual memory fragmentation, the
msExch ESEParamMaxOpenTables attribute in Active Directory would be lowered. Typically this change was made only at the direction of Microsoft PSS; however, it was a documented value, so many administrators have made this change. Certainly you'll want to check your own storage groups to see if the value exists. It was often used in tandem with the /3GB switch in the Windows BOOT.INI file, so it's a pretty safe bet that if you are using /3GB, you probably also have this value set.
msExchESEParamMaxOpenTables is an attribute of storage groups. The recommended value for this attribute also changed periodically throughout the life of Exchange 2000. In Service Pack 2, the default value was automatically set to 42500 on four-way systems and 85000 on eight-way systems. In Service Pack 3, this was changed to 13800 and 27600, respectively. If you do find a value set for
msExchESEParamMaxOpenTables on any Exchange 2000 storage group, regardless of its value you will want to return the value for this attribute back to <Not Set> prior to upgrading to Exchange 2003.
Content expiration in Outlook Web Access
Like Exchange 2000 OWA, the Exchange 2003 version of OWA is comprised partially of static files (such as image files, scripts, and so forth). Typically these files were changed only when administrators either customized one or more OWA files or installed an Exchange service pack. Despite the fact that the files remained unchanged for long periods, they were marked as expiring one day after being fetched by the Web browser. Because the files expired each day instead of being retrieved from the browser cache, they were pulled down every day by every user from the OWA server—the same file, the same user, different days. To stop this insanity, Microsoft recommended that the virtual directory that served the static files— Exchweb—be configured with a much longer content expiry (a year or so).
While this method worked to reduce the load on your network and your OWA server in Exchange 2000, it doesn't work in Exchange 2003. In fact, on Exchange 2003 servers, the Exchweb virtual directory should always have its content expiration set to 1 day. It should not be disabled, and it should not be set to anything greater than 1 day.
Exchange Server SMTP service tuning
On busy Exchange 2000 systems that sustained large SMTP message queues (e.g., an average of 1,000 or more), performance constraints were encountered because of a default setting on the SMTP service of a maximum of 1,000 file handles. Each time the SMTP transport stack on an Exchange 2000 (or Exchange 2003) server receives a message, it is streamed out to the file system, where it waits to be routed to its destination. To write it to the file system, the SMTP transport stack obtains a file handle and then passes the message into that handle. Because Exchange 2000 defaulted to a maximum of 1,000 file handles, the SMTP service could write only 1,000 simultaneous messages to the file system.
To improve performance for these large systems, three registry entries were often simultaneously adjusted to increase the maximum number of file handles that could be opened by the SMTP service (so that more messages could be processed) and to decrease the number of open file handles for the installable file system, another Exchange component (to avoid running out of memory when the queue is large). These registry values, which did not exist by default and therefore needed to be added manually, are listed here.
MsgHandleAsyncThreshold entries would be set to the same value (some value greater than 1000), and the
FileCacheMaxHandles entry would be reduced from 800 to 600.
Exchange 2003 dynamically calculates the appropriate settings for SMTP files handles, so these settings are no longer needed. Therefore, before upgrading any Exchange 2000 servers with these settings to Ex¬change 2003, you should delete the entries from the registry.
7 tips in 7 minutes: Exchange Server 2003 tips and tricks
Tip 1: Tuning Exchange Server 2003 overview
Tip 2: Exchange 2000 vs. Exchange 2003 tuning parameters
Tip 3: Exchange 2003 tuning parameters -- Outlook Web Access
Tip 4: Exchange 2003 tuning parameters -- Microsoft Outlook
Tip 5: Exchange 2003 tuning parameters -- Exchange Server
Tip 6: Must-have Exchange Server 2003 tools
Tip 7: Exchange Server administration resources and links
|This chapter excerpt from Microsoft Exchange Server 2003 Distilled by Scott Schnoll, is printed with permission from Addison-Wesley Professional, copyright 2004.|
Dig Deeper on Microsoft Exchange Server Performance