During MEC 2014 I delivered a presentation on “Backup, Restore and Disaster Recovery” in Exchange 2013. The first part of this presentation was about Mailbox databases and its accompanying transaction log files. In this blog I’ll explain a bit how to replay these transaction log files is a recovery scenario.

Please not that this is only viable for a Mailbox server that’s not a member of a DAG. When you have a DAG this technology is getting less important for recovery purposes you still have another running copy in the DAG. However, when you want to replay a lagged copy this information can be useful.

Mailbox Databases and Transaction Log Files

I won’t go into too much detail about the internal Mailbox database technologies (see my Simple Talk article on http://bit.ly/DBSimple) but a Mailbox data is located in three places:

• Server Memory – This is where Mailbox data is cached and where new transactions are created;
• Transaction Log Files – This is where all transactions are stored before they are flushed to the Mailbox database. All transactions are always stored in the Transaction Log Files;
• The Mailbox dabases – This is where the actual Mailboxes are stored and where all Mailbox data resides.

It’s a dynamic system and Mailbox data can be located in one of three places. Transaction log files are always up-to-date with the latest transactions, but the newest information might not be flushed to the Mailbox database but still is in Server memory. The information that’s newer than the checkpoint file at a certain point in time is still in Server memory but not flushed to the Mailbox database. However, this information is safely stored in the latest transaction log files as shown in the following figure:

When a database is running its status is “not consistent” or “dirty shutdown”. This means the data is spread across all three locations. When a Mailbox database is dismounted all information is flushed to the Mailbox database (which means all information in the latest transaction log files is now in the Mailbox database) and the Mailbox database is closed.

Important to note is that all transactions will be stored in the transaction log files. So the delivery of new messages, calendar items, tasks, creation or deletion of folders or mailboxes, everything is stored in the transaction log files. This can be used for reccovery purposes. If you have all the log files available it is even possible to reconstruct the entire Mailbox database from the transaction log files.

The checkpoint file determines where recovery will begin. Exchange thinks that all information in transaction log files ‘below’ the checkpoint file will be safely stored in the Mailbox database, thus recovery will begin at the location indicated by the checkpoint file. If the checkpoint file is no longer available recovery will start with the oldest available transaction log that’s on the Exchange server.

Let’s look at the following example. We have an Exchange server that’s running for some time and messages are stored in the Mailbox database. We can logon to the Mailbox, send and receive messages, created new meetings etc.

If we dismount the Mailbox dabase (and thus bringing the Mailbox database into a consistent state) we can check the state of the Mailbox database using the following command:

ESEUTIL /MH MDB01

You’ll see the the State is “Clean Shutdown” which indicated the Mailbox database is properly shutdown. The Log Required field is zero and so is the Log Commited. Since this is a fairly new Mailbox database all transaction log files (starting with E0000000001.log) are available.

When we delete the Mailbox database and try to recover the Mailbox database we can use the following command:

ESEUTIL /R E00 /lF:\MDB01\Logfiles /sF:\MDB01\Logfiles

The /l indicates the location of the system files (i.e. the checkpoint file) and the /s indicated the location of the transaction log files.

However, no Mailbox database is constructed. What’s wrong here is that the checkpoint file is still available. This means that recovery will start at that point, but since the Mailbox database is properly dismounted it will point the last available location so there’s nothing to recovery.

To successfully reconstruct everything the checkpoint file needs to be deleted as well. We we do and retry the ESEUTIL command we get a better result:

So, when we lose the Mailbox database but when we still have all the transaction log files available it’s not a big deal. We can reconstruct the Mailbox database, mount it again and logon to the Mailboxes.

When you check the eventlog you’ll see an entry from ESE (Event ID 325) indicating that a new Mailbox database was created. The creation of the original Mailbox database was also stored in (of the first) transaction log files. Since the log file is replayed the Mailbox creation transaction is replayed as well.

When checking the state of the Mailbox database after recovering we should see the same information as before:

Now the Create time entry you can see in the screen shot above is dated March 27, while the actual replay of the transaction log files took place on April 1^st. The date as shown above is the date of the original Mailbox database file, when the actual transaction was performed and stored in the log file. Combined with the timestamp and the random number it makes a unique identifier for this particular Mailbox database file.

We can also check the transaction log files. When we navigate to the directory where the transaction log files reside we can enter the following command:

ESEUTIL /ML E00

This will loop sequentially through all transaction log files with the prefix E00 and check them as shown in the following figure (where the command was aborted using CTRL-C):

Replaying transaction log files is only possible when all transaction log files are available. For example, when we delete a number of transaction (using explorer as shown in the following figure) and try the ESEUTIL /ML command again it will show the missing log files:

So right now there’s a gap in the number of available transaction log files, comparable to a missing incremental backup in a series of incremental backups.

When we try to run the ESEUTIL /R command to recovery the Mailbox database it will fail at the last available transaction log file in the sequence:

When we check the status of the Mailbox database using the ESEUTIL /MH command we can see the database is not in a consistent state:

Now you might notice the first log file in the Logs Required section is availble on disk (check the Windows explorer screenshot) but that’s because the transactions are not distributed sequentially in the transaction log files.

There’s no way to work arount the missing log files. The only option is to tell Exchange that the latest available transaction log file in the first sequence (E0000000FF.log) is the last available log file. This is achieved by renaming it to E00.log which means the last available log file for the Mailbox database.

Now when we run the ESEUTIL /R command again it will replay the transcation log files, but it will stop at the last available log files. All newer transaction log files will never be replayed. This automatically means that all information in these log files gets lost!! It therefore takes less time to recover the Mailbox database.

Now when we mount the Mailbox database we can login to the Mailbox and see that we have missing messages in the inbox!

In this example we only had to reconstruct a 120MB Mailbox database which took some time (this time depends on the hardware being used though) but imaging you need to reconstruct a Mailbox database that’s a couple of hundreds Gigabytes. This will takes quite some time.

Offline backup and restore

An offline backup is something I would never recommend to customers since it is a lot of work, prone to error and rather complex to maintain. Also you have to be an expert on Mailbox databases and recovery, and it’s something most customers don’t fancy. Oh, and since it is an offline backup you need to dismount a Mailbox database which means a service disruption.

What happens during an offline backup is that you dismount the Mailbox database, copy the Mailbox database to a safe location, remove the older log files (not all, but you can check the Mailbox database header information using ESEUTIL /MH to determine which ones are safe to delete) and continue working.

When you need to restore, you can copy the backup file to the original location and replay the newer transaction log files and get the data into the original Mailbox database file.

So, after dismounting the Mailbox database, copying it to a safe location we can check the Mailbox database header information using ESEUTIL /MH as shown in the following figure:

The transaction log file used when dismounting was E00000001D2.log as shown in the Last Detach entry. So, earlier transaction log file can be removed, but just to be safe we leave a couple of other transaction log files on the server as well.

It is extremely important NOT TO DELETE ALL LOG FILES. If you do and mount the Mailbox database a new set of log files will be created. These are not compatible with the previous set of log files (although the name looks the same) and you’re not able to recover information the way you want!

Now mount the Mailbox database and continue working. Over time (lots of) new transaction log files will be generated, all containing new information for this particular Mailbox database. If you want to create incremental backup using this method you can move older transaction log files to a safe location. It is not needed to dismount the Mailbox database, but you have to be very accurate when moving transaction log file. If one log file is missing the recovery using ESEUTIL /R will fail as we have seen earlier in this chapter.

Now dismount the Mailbox database and delete the MDB01.EDB file, pretending it was crashed beyond repair. To restore, copy the file that was copied earlier to a safe location to its original location. Do not forget to remove the checkpoint file as Exchange uses this to determine what to do. If you do not remove it Exchange will not recover because it thinks that all information is in the Mailbox database (and as we know, it’s not). If you have some additional transaction log files as part of an incremental backup you can copy these back to the original location as well. Make absolutely sure you are not missing any transaction log file or the process will halt somewhere during recovery.

To recover the information is the remaining transaction log file you can use the ESEUTIL /R again:

ESEUTIL /R E00 /sF:\MDB01\Logfiles /lF:\MDB01\Logfiles

Since only a small amount of log files need to be replayed it doesn’t take that long. After you mount the Mailbox database and login to a Mailbox you’ll see that information that was received AFTER the last backup was created is again in the Mailbox.

What happens if your Mailbox database contains corrupt pages? In this case nothing happens. The Mailbox database is stored safely on another location and so are the corrupt pages. You’ll never know they are until you check the Mailbox database for consistency using the following command:

ESEUTIL /K MDB01.EDB

As you can see this Mailbox database is perfectly healthy.

Summary

In this post (and during my presentation) I showed you that you can recover a Mailbox database when it gets lost. I was able to recover from this, but the prerequisite is that all transaction log files from the beginning are available.

If you want to use an offline backup (which I don’t recommend unless you’re an expert on this and have lots of spare time) you have to dismount the Mailbox database, copy the EDB file to a safe location, check the Mailbox database for consistency and manually remove the older transaction log files (but not all of them).

I wrote this down so you can see what’s actually happening under the hood (from a logical perspective) when restoring a backup and replaying information from the transaction log files. But remember, it is far from complete so I would not use it in a production environment 🙂

In my next blog I’ll discuss the ‘real’ backup methods using VSS.