As you scroll through the Application Log, look for which diagnostic logging was initially enabled. Once you find this, look for Event IDs 8011 and 8012. The Application Log may contain a large number of log entries, so it might be difficult to find these Event IDs. If you have trouble locating them, use the Filter option located on the Event Viewer's View menu.
Event ID 8011 and Event ID 8012 indicate that the Exchange Recipient Update Service (RUS) has started. If you are unable to locate these Event IDs, then the RUS has either not started or is not responding.
What is an 8011 event?
An 8011 event is nothing more than an LDAP query. When you first start Exchange Server diagnostic logging, there will be a long series of 8011 events, followed by 8012 events. As you go through these log entries, look for a very specific 8011 event. On my test server, the 8011 event of interest was the seventh 8011 event recorded; however, the actual sequence may vary. The event should look something like Figure 5.
Figure 5. Here is an example of an 8011 event.
In Figure 5, you'll notice that the description portion of the window contains a lengthy LDAP query string. But in this string, there is only a small portion that is of interest.
This query string searches for objects that belong to the msExchangeAddressListService Class, but don't have their isDeleted attribute set to True. This means that the query is searching for any Recipient Update Service objects that have not been tombstoned.
Tombstoned objects are objects that have been deleted, but that have not yet been removed from Active Directory. Therefore, this particular query string is searching for RUS objects that have not been deleted.
Immediately following this 8011 event, there should be an 8012 event that displays query results (Figure 6). Notice that the event doesn't list specific objects that the query returns. Instead, it tells you that the query has returned two objects because there were only two RUS objects in the Exchange Server organization. These are the Enterprise RUS and a domain RUS.
Figure 6. Event 8012 shows you how many RUS objects were located.
Don't ignore 8012 events
When troubleshooting an Exchange Recipient Update Service problem, there is often a temptation not to look at the description of Event ID 8012 to save time. But this log is extremely important because you must know how many RUS objects were located.
Suppose, for instance, that no objects were located. The 8011 event would appear exactly as it does in Figure 5. Likewise, the 8012 event would appear immediately after that. If you don't take time to look at the 8012 event's description though, you are missing an important piece of information. In addition, if no objects were found, the 8011 and 8012 events would be repeated frequently to locate the RUS.
What happens if you can't locate these two events in your logs? If the Exchange server is a front-end server, it is normal for these two events not to occur. If the Exchange server is a back-end server, and you were unable to locate 8011 and 8912 events, try restarting the Exchange System Attendant service. If you are still cannot locate the events, most likely there is a problem with the ABV_DG.DLL file, whose job it is to search for the RUS.
Understand how Event IDs sequence
To further troubleshoot this problem, you must understand the sequence of events that occurs when you start the Exchange System Attendant service during diagnostic logging. To troubleshoot the startup sequence, look for Event ID 1000, which should be relatively easy to locate because it is not generally a recurring event. Event ID 1000 simply informs you that the Exchange System Attendant service has started successfully.
The next log entry you want to look for is Event ID 9006. There could potentially be some other events between 1000 and 9006 that relate to other services starting. This is especially true if you reboot your server, rather than restarting the Exchange System Attendant service, as shown in Figure 7.
Figure 7. There may be other events between 1000 and 9006.
Event 9006 is produced any time the MSExchangeSA service loads a DLL file. If you look at Figure 7, you will notice that Event ID 9006 occurs several times. You must look at all event log entries until you locate the reference to the ABV_DG.DLL file. Figure 8 gives an example of this event.
Figure 8. Event ID 9006 indicates that the ABV_DG.DLL file is loading.
Event ID 9006 indicates that the ABV_DG.DLL file is loading, but it never says anything about it loading successfully. Event ID 9008 indicates that the ABV_DG.DLL file is starting, as shown in Figure 9. This is important because unless this event occurs, the 8012 event that I discussed earlier will always return zero results.
Figure 9. Event number 9008 confirms that the ABV_DG.DLL is starting.
As you search for Event ID 9008, remember that every 9006 event should have a corresponding 9008 event. So if you have several 9006 events, then you also have to look through all the 9008 events until you find the one that references the ABV_DG.DLL file.
Find the ABV_DG.DLL file
In the startup sequence, everything depends on the ABV_DG.DLL file. Unless this file is loaded and functional, Exchange Server will not be able to locate Recipient Update Service objects within Active Directory.
Consider the implications of receiving a 9006 event without a corresponding 9008 event. According to Microsoft, the only time you should see a 9006 event (related to the ABV_DG.DLL file) without a 9008 event is when Exchange is set up as a front-end server.
Because a 9008 event indicates that the ABV_DG.DLL file is beginning, having a 9008 event without an 8011 event represents a serious problem. If an 8011 event doesn't follow the 9008 event, then the DLL file either didn't start or has stopped responding.
In such a situation, it could mean that the ABV_DG.DLL file is damaged. You might try fixing the problem by reapplying the service pack to the server or stealing a copy of the ABV_DG.DLL file from another Exchange server running at the same service pack level.
If Event IDs 1000, 9006, 9008 and 8011 occur as expected, but Event ID 8012 indicates that no objects are found, look for an Active Directory communications problem. Make sure that your Exchange Server can communicate with your domain controllers and that DNS queries are working correctly. You can test this with the Nslookup command. Also verify that your global catalog server is online and accessible.
TROUBLESHOOTING THE RECIPIENT UPDATE SERVICE
Step 1: Validate server names for the Exchange Recipient Update Service
Step 2: Diagnose DNS-related Exchange Recipient Update Service problems
Step 3: Enable diagnostic logging for the Exchange Recipient Update Service
Step 4: Find Exchange Recipient Update Service Event ID log entries
Step 5: Troubleshoot the Exchange RUS with ADSI Edit and 8011 events
|ABOUT THE AUTHOR:|
| Brien M. Posey, MCSE
Brien M. Posey, MCSE, is a Microsoft Most Valuable Professional for his work with Exchange Server, and has previously received Microsoft's MVP award for Windows Server and Internet Information Server (IIS). Brien has served as CIO for a nationwide chain of hospitals and was once responsible for the Department of Information Management at Fort Knox. As a freelance technical writer, Brien has written for Microsoft, TechTarget, CNET, ZDNet, MSD2D, Relevant Technologies and other technology companies. You can visit Brien's personal Web site at http://www.brienposey.com.