Speed up mailbox migration to Exchange 2013 – They’re too slow!

Moving mailboxes from Exchange 2007 or 2010 to Exchange 2013 can often go very slowly, even when the network and server resources are fast and abundant!  The Exchange Mailbox Replication Service (MRS) has extensive resource throttling enabled by default in order to prevent mailbox moves from choking out the rest of the users.  Because of this you may see mailboxes with a status of RelinquishedWlmStall and if you look at the details of the Get-MoveRequestStatistics report you will see mailboxes have a lot of time sitting idle under the TotalStalledDueToWriteThrottle counter. Microsoft tech support suggests making changes to the “MSExchangeMailboxReplication.exe.config” file located at “C:\Program Files\Microsoft\Exchange Server\V15\Bin”.  The values to look at, along with their default settings are: MaxActiveMovesPerSourceMDB=”20″ MaxActiveMovesPerTargetMDB=”20″ MaxActiveMovesPerSourceServer=”100″ MaxActiveMovesPerTargetServer=”100″ MaxTotalRequestsPerMRS=”100″ ExportBufferSizeKB=”512″ We typically like to set these values so that about 10 mailboxes can be moved simultaneously.  The ExportBufferSizeKB we’ve used in the past is “10240”.  The Exchange Mailbox Replication Service should be restarted after these changes. The other suggestion Microsoft has made is to disable content indexing on the target database so that the search index scanner isn’t overwhelmed by all the new messages needing to be indexed.  You’ll want to set it back once the migration is complete. Set-MailboxDatabase “DB1” -IndexEnabled:$False In our experience however, these first two suggestions do NOT have tremendous impact on the overall speed.  The following two options have proven to be the most effective for us. Use the “-priority emergency” parameter on the mailbox moves.  This will give the move the highest priority in the MRS queue.  For example: New-MoveRequest -Identity “user@domain.com” -TargetDatabase “DB1” -Priority emergency If the priority flag and the MRS config editing doesn’t make the moves fast enough for you, then disable MRS throttling altogether!  To...

Exchange 2013 DAG: Can’t add a database copy – “Seeding operation failed”

We had an issue when attempting to add a database copy for any mailbox databases in a new DAG.  We received one of the following errors every time, regardless of the source or destination server. The seeding operation failed. Error: An error occurred while running prerequisite checks. Error: The specified database isn’t configured for replication and therefore cannot be used to perform seed operations. The seeding operation failed. Error: An error occurred while performing the seed operation. Error: An error occurred while processing a request on server ‘MailboxServerName’. Error: Database ‘6060c9ac-363a-4e52-a02e-ba749625e8ea’ was not active on source server ‘MailboxServerName’. [Database: Test3, Server: MailboxServerName.domain.com] After closing out the dialog box we found that the database showed multiple copies, but that the copy was in a failed and/or suspended state. We troubleshot this and found that the client’s several domain controllers were not replicating information as quickly as we were used to.  To help with this, we set the preferred domain controller to a global catalog DC residing in the same network using the following command:  Set-ADServerSettings -PreferredServer server.domain.com This seemed to help some other minor ECP errors where we felt like IIS was thinking faster than AD could keep up, but this did not remove the errors we saw when creating a database copy. The solution was to walk away and wait for about 30 minutes and on return we found that some of the mailboxes cleaned up on their own and went to a healthy state.  For the majority that didn’t resolve themselves we fixed them in ECP by selecting the database and on the right pane click “Update” under the database copy that failed. In the dialog box that followed we did not specify a source server, continued, and the mailbox replicated...

Exchange 2013 – Can’t access ECP, 500 Unexpected Error

We had an issue with a client migrating from Exchange 2007 to 2013 where we couldn’t access ECP on any of the new 2013 CAS servers.  Instead we received the following: 500 Unexpected error  : (  An error occurred and your request couldn’t be completed. Please try again. We also noticed that although the Exchange Management Shell would successfully open and operate, on first opening it we received an error that contained the following message: Unable to determine the installed file version from the registry key ‘HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\PowerShell\1\PowerShellEngine’.  After much troubleshooting (reinstalling CAS servers, deleting/recreating ECP folders, checking permissions, etc.) we found that we needed to install the “Windows PowerShell 2.0 Engine” feature from Server Manager (this also installed .NET 3.5).  These features needed to be installed on ALL of the new 2013 Exchange servers and not just the CAS servers. Although these features aren’t a prerequisite for a 2007 to 2013 migration, there must’ve been something in the existing AD/Exchange environment that required it.  Hope this helps someone...

Trouble with Exchange UM and arbitration mailbox corruption

One of our clients had an issue where their Unified Messaging arbitration mailbox had become corrupted.  We were unable to change the audio files on the auto-attendants and we were seeing errors like this in the application logs: Event ID 1431 UM can’t save the Call Data Record message to the EDiscovery mailbox. More information “The number of call data record messages being processed in the Unified Messaging pipeline has exceeded the maximum limit. Please check whether the EDiscovery mailbox with DistinguishedName CN=SystemMailbox{e0dc1c29-89c3-4034-b678-e6c29d823ed9},CN=Users,DC=domain,DC=com or MailboxDatabase CN=UM-Mailbox-DB,CN=Databases,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=domain,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=domain,DC=com is down.”. Event ID 1099 User “SYSTEM” attempted to modify a custom prompt for auto attendant “AA 001”. The operation failed. Make sure that the user is assigned the UM Prompt role or UM Management role. More information: “An error occurred while accessing the custom prompt publishing point. The underlying connection was closed: An unexpected error occurred on a send.”. After unsuccessful attempts to repair the arbitration mailbox we deleted and recreated the mailbox.  First we removed the arbitration mailbox: Disable-Mailbox “SystemMailbox{e0dc1c29-89c3-4034-b678-e6c29d823ed9}” -Arbitration Note: We did not opt to delete the user account associated with the UM mailbox in this particular case.  If both the UM arbitration mailbox and user account are somehow corrupted then it may also be necessary to delete the “SystemMailbox{e0dc1c29-89c3-4034-b678-e6c29d823ed9}” user account under the AD Users folder after disabling the mailbox.  To recreate that user run “setup.exe /PrepareAD” from the Exchange installation folder and the user will be created again. Then we recreated the mailbox: Enable-Mailbox -Arbitration -Identity “SystemMailbox{e0dc1c29-89c3-4034-b678-e6c29d823ed9}” Then we set the display name to the correct value “Microsoft Exchange”, because the default setting was wrong: Set-Mailbox -Arbitration -Identity “SystemMailbox{e0dc1c29-89c3-4034-b678-e6c29d823ed9}” -DisplayName...

Changing the URLs and names for an Exchange Server

If your Outlook clients are complaining about a cert name mismatch or if you’re having other authentication issues then you might need to check that the names being presented match the cert.  Here are some common places to check: Get-ClientAccessServer – AutoDiscoverServiceInternalUri Get-AutodiscoverVirtualDirectory – InternalUrl, ExternalUrl Get-OutlookAnywhere - ExternalHostname Get-OutlookProvider - EXCH: CertPrincipalName,  EXPR: CertPrincipalName Get-WebServicesVirtualDirectory - InternalUrl, ExternalUrl Get-OwaVirtualDirectory - InternalUrl, ExternalURL Get-OabVirtualDirectory - InternalUrl, ExternalUrl Get-EcpVirtualDirectory - InternalUrl, ExternalUrl Get-ActiveSyncVirtualDirectory - InternalUrl, ExternalUrl TIP: As a shortcut, just copy and paste the following lines into powershell and examine the results! Get-ClientAccessServer | fl Identity,AutoDiscoverServiceInternalUri Get-AutodiscoverVirtualDirectory | fl Identity,InternalURL,ExternalUrl Get-OutlookAnywhere | fl identity,ExternalHostname Get-OutlookProvider | fl identity,name,CertPrincipalName Get-WebservicesVirtualDirectory | fl Identity,InternalURL,ExternalUrl Get-OwaVirtualDirectory | fl Identity,InternalURL,ExternalUrl Get-OabVirtualDirectory | fl Identity,InternalURL,ExternalUrl Get-EcpVirtualDirectory | fl Identity,InternalURL,ExternalUrl Get-ActiveSyncVirtualDirectory | fl Identity,InternalURL,ExternalUrl...

Exchange 2013 and Windows XP with wildcard certs

We had a client that upgraded from Exchange 2007 to 2013.  After the migration the Windows XP machines would not connect to Exchange.  The users were presented with a credentials / password pop-up repeatedly and no matter what was entered the Outlook client would never connect to Exchange. Having dealt with XP in other Exchange 2013 environments we tried all the usual tricks: Update Office 2007 / 2010 to the correct hotfix level Edit the lmcompatibility level in the registry of the XP box by locating the following registry key:  HKEY_LOCAL_MACHINESYSTEMCurrentControlSetControlLsa Change lmcompatibilitylevel object to 2 or 3 (we used 3), then restart computer. Run “Set-OutlookProvider EXPR -CertPrincipalName msstd:mail.domain.com” Manually set the security in the Outlook client to NTLM. Change the OWA authentication methods to the following: InternalHostname                   : email.domain.com ExternalClientAuthenticationMethod : Negotiate InternalClientAuthenticationMethod : Ntlm IISAuthenticationMethods           : {Basic, Ntlm, Negotiate} But none of these worked!  The unique thing about this customer’s setup compared to others was that they were using a wildcard cert.  We noticed that the cert SAN name for OutlookProvider was set to “mail.domain.com” and it really should’ve been “*.domain.com”.  Here is the command that saved the day. Set-OutlookProvider -Identity EXCH -CertPrincipalName msstd:*.domain.com Do a “get-OutlookProvider | fl” in order to confirm the settings.  Then wait for a few minutes and try again.  We had to open Outlook once with a failure to login and then close Outlook and start it again.  The continual login prompt was gone! Side note: For the Set-OutlookProvider command EXCH is for internal OWA clients and EXPR is for external...