|
I recently received an e-mail from an administrator in a bad situation. He had an Exchange 2003 server with a corrupted mailbox store. He wanted to repair the store, but was hesitant to do so for a couple of reasons.
For starters, he knew that if his mailbox store contained corrupted data, then his backups probably did too. This being the case, there was no guarantee that he would be able to restore the backups if something went wrong.
His was also worried about the unpredictable nature of ESEUTIL. When ESEUTIL is used to repair a corrupted database, it deletes anything it doesn't understand. Going in, there's no way to know how much data is going to be lost.
Testing recovery
Here's what I advised him to do:
- Set up a test network not connected to the production network.
- Restore the backup of a domain controller -- that is also functioning as a DNS server and global catalog server -- onto one of the test network servers. (Since Exchange is dependent on Active Directory, having this server on the test network allows Exchange to function as it would in a production environment.)
- Restore a backup of the Exchange server to a server on the test network.
Running a repair against a database on a test server isn't going to give you the exact same results you would get if you ran the process against the production server, because the production server's database is constantly being modified as users interact with it.
But the results should be similar enough to give you a good feel for what will happen when you attempt the repair process in your production environment.
If your recovery test fails
The above procedure works great if you just need to verify that your backups work or that ESEUTIL isn't going to destroy your databases. Let's take this a step further though. Suppose that the restore operation failed or ESEUTIL obliterated the database. What then? Should you just leave well enough alone, or should you attempt a recovery anyway?
Obviously, that's a tough call to make and you will have to use a little common sense based on what you know about your individual situation. After all, no two servers are exactly alike. In my opinion though, just leaving a corrupted database alone is probably a bad idea over the long term. Exchange isn't going to heal itself, and the problem could gradually get worse, especially if an update requires modification of the database schema.
In a situation like that, here's what I would do:
- Bring a temporary Exchange server online as a part of your production Exchange organization. Make sure that it's in the same Active Directory site and routing group as the failing Exchange server, and that it has enough disk space to accommodate all the data from the failing Exchange server.
- Use the Move Mailbox Wizard to move mailboxes off of the corrupt server and onto the temporary server.
The Move Mailbox Wizard contains an option for skipping corrupted items within a mailboxes and creating a failure report. The failure report is basically just a log of which items have been skipped. By default, the Move Mailbox Wizard will not move a mailbox if it contains more than three corrupted items, but the wizard allows you to adjust this number.
- After you move all of the good mailboxes and their contents to the temporary server, nothing should be left within the store except for corrupted items. At this point, take a break and give everything time to replicate.
- Then, take the corrupt store offline and run ESEUTIL against it to repair the store. You can do this without fear because all your good data is safely stored on the temporary server.
- Once ESEUTIL completes, run it one more time to verify that the corruption is gone.
- Finally, mount the store and move all of the mailboxes back to it.
Once the server is back to a functional state, you should immediately make a backup.
Brien M. Posey, MCSE, is a Microsoft Most Valuable Professional for his work with Windows 2000 Server and IIS. Brien has served as the CIO for a nationwide chain of hospitals and was once in charge of IT security for Fort Knox. As a freelance technical writer he has written for Microsoft, TechTarget, CNET, ZDNet, MSD2D, Relevant Technologies and other technology companies. You can visit Brien's personal Web site at www.brienposey.com.
MEMBER FEEDBACK TO THIS EXCHANGE SERVER TIP
This is all straightforward as long as you can mount the store. Without the store mounted, the options are a lot more limited.
—Richard L.
******************************************
I just ran into this exact same problem over the past week on our Exchange 2000 server. I discovered through the event logs that the Exchange mailbox store contained corrupted data. The problem was that the event log was so full of events, I could only go back three days. Therefore, I could not determine when the problem first occurred (the errors in the event log did span those three days).
As a result, I couldn't use an online backup since I would most likely just be restoring a corrupted database. I contacted Microsoft, and after taking some time to determine the best way to approach it, we ended up doing the following procedure:
- Used Microsoft's ExMerge utility to export all mailboxes into a .PST file. This process will only export any "good" data and leave the corrupted data behind.
- Dismounted the mailbox store, renamed the MDBDATA directory that contained the databases to MDBDATA.OLD, and created a new folder called MDBDATA.
- Mounted the mailbox store, which created a new blank database.
- Used the ExMerge utility to import all of the mailboxes from the .PST files.
There were some steps in between, but this is primarily what it took to get our Exchange server back up and running. It seems to have worked better than I expected, with no complaints from users of any loss of data. Of course, results could have been different if the corruption was larger. However, I think this is a much easier method than some of the alternate solutions.
—Mark S.
******************************************
"... nothing should be left within the store except for corrupted items."
That's not exactly accurate in all situations. If there are more than 100 corrupted items in each mailbox (the upper limit on corruption before Move Mailbox Wizard fails), you will have mailboxes left in the store with far more valid data than corrupt data. This could happen if your priv1.stm gets accidentally deleted and rendered unrecoverable.
—Travis S.
******************************************
This is still a very valid article. I just fixed a similar issue on Windows SBS and Exchange Server 2003 using the instructions provided by member Mark S. It saved the day.
—Matthew A.
******************************************
Your article was very useful for my situation, but I have a question. Will the old database (i.e., the corrupted mailbox store) corrupt other databases on the same Exchange server or even in same storage group?
—Vijay P.
******************************************
Wow, that's a tough question. It depends on the cause of the database corruption. Everything in the databases is initially written to the transaction logs before it is written to the database. If the problem affects the integrity of the transaction logs, then multiple databases can be affected, especially within a storage group. Typically a situation like that would be caused by faulty memory or by a faulty disk controller.
—Brien Posey, tip author
Do you have comments on this tip? Let us know.
Please let others know how useful this tip is via the rating scale below. Do you have a useful Exchange Server or Microsoft Outlook tip, timesaver or workaround to share? Submit it to our tip contest and you could win a prize.
|
Rate this Tip
|
To rate tips, you must be a member of SearchExchange.com.
Register now
to start rating these tips. Log in if you are already a member.
|
Submit a Tip
|
DISCLAIMER: Our Tips Exchange is a forum for you to share technical advice and expertise with your peers and to learn from other enterprise IT professionals. TechTarget provides the infrastructure to facilitate this sharing of information. However, we cannot guarantee the accuracy or validity of the material submitted. You agree that your use of the Ask The Expert services and your reliance on any questions, answers, information or other materials received through this Web site is at your own risk.
|
|
|
