I am honored to be a speaker at the Virtual Scottish Summit 2021 conference, taking place on Saturday, 27. February.

The Scottish Summit is a truly European event. You can choose from 365 sessions covering mostly any about Microsoft 365 workloads in seven languages:

• Englisch
• Spanish
• German
• French
• Italian
• Portuguese
• Polish

Attend my session when you are interested in the challenges of implementing Exchange Server Hybrid, and the requirements to make it work with Microsoft Teams and on-premises mailboxes.

My session at Scottish Summit 2021:

• Exchange Hybrid - What, Why, and How
• Session schedule details: TBD

Links

• Scottish Summit 2021 Website
• Register for Scottish Summit
• Scottish Summit 2020 YouTube Channel

See you online for the Virtual Scottish Summit.

You sometimes need some (or even many) test user objects in Active Directory.

This script helps you create any number of test users in your Active Directory domain, which you can easily enable for on-premises or remote mailboxes afterward.

The Script

# Number of user accounts to create
$UserCount = 5
$RandomPassword = $true
$DefaultPassword = 'Pa55w.rd'

# User name prefix
# New user object will be named TestUser1, TestUser2, ...
$TestUserPrefix = 'TestUser'

# User object properties
$GivenName = 'Test'
$Surname = 'User'
$Company = 'Varunagroup'
$JobTitle = @('Junior Consultant','Senior Consultant','Technical Consultant','Business Consultant')
$PreferredLanguage = 'de-DE'

# Name of the new organizational unit for test user object
$TestOU = 'Test User'

# Target OU path where the script creates the new OU 
$TargetOU = 'OU=IT,dc=varunagroup,dc=de'

# Import Active Directory PowerShell Module
Import-Module -Name ActiveDirectory 

# Build OU Path
$UserOUPath = ("OU={0},{1}" -f $TestOU, $TargetOU)

# Check if OU exists
$OUExists = $false

try {
   $OUExists = [adsi]::Exists("LDAP://$UserOUPath")
}
catch {
   $OUExists =$true   
}

if(-not $OUExists) { 
   # Create new organizational unit for test users
   New-ADOrganizationalUnit -Name $TestOU -Path $TargetOU -ProtectedFromAccidentalDeletion:$false -Confirm:$false
}
else {
   Write-Warning ('OU {0} exists please delete the OU and user objects manually, before running this script.' -f $UserOUPath)
   Exit
}

Write-Output ("Creating {0} user object in {1}" -f $UserCount, $UserOUPath)

# Create new user objects
1..$UserCount | ForEach-Object {

   # Get a random number for selecting a job title
   $random = Get-Random -Minimum 0 -Maximum (($JobTitle | Measure-Object). Count - 1)

   # Set user password
   if($RandomPassword) {
      # Create a random password
      $UserPassword = ConvertTo-SecureString -String (-join ((33..93) + (97..125) | Get-Random -Count 25 | % {[char]$_})) -AsPlainText -Force
   }
   else {
      # Use a fixed password
      $UserPassword = ConvertTo-SecureString -String $DefaultPassword -AsPlainText -Force
   }

   # Create a new user object
   # Adjust user name template and other attributes as needed
   New-ADUser -Name ("{0}{1}" -f $TestUserPrefix, $_) `
   -DisplayName ("{0} {1}" -f $TestUserPrefix, $_) `
   -GivenName $GivenName `
   -Surname ("$Surname{0}" -f $_) `
   -OtherAttributes @{title=$JobTitle[$random];company=$Company;preferredLanguage=$PreferredLanguage} `
   -Path $UserOUPath `
   -AccountPassword $UserPassword `
   -Enabled:$True `
   -Confirm:$false
}

Enable mailboxes

Use your on-premises Exchange Management Shell to enable all test users with an on-premises mailbox.

$UserOU = 'OU=Test User,OU=IT,dc=varunagroup,dc=de'
Get-User -OrganizationalUnit $UserOU | Enable-Mailbox -Confirm:$false

Use your on-premises Exchange Management Shell to enable all test users with a new remote mailbox in Exchange Online. Do not forget to change the tenant name of the remote routing address.

Get-User -OrganizationalUnit 'OU=Test User,OU=IT,dc=varunagroup,dc=de' | %{Enable-RemoteMailbox
 -Identity $_ -Confirm:$false -RemoteRoutingAddress "$($_.SamAccountName)@TENANT.mail.onmicrosoft.com"}

You find the most recent version of the script at GitHub.

Links

• GitHub: https://github.com/Apoc70/New-TestUser

Enjoy.

Services of third-party software solutions often interfere with installing a new Exchange Server cumulative update, because these services have a file lock active. 

To avoid any issues when installing a CU, or having the prerequisites check fail due to open files, you simply stop the Windows services and ensure that those services do not restart automatically. Especially monitoring solutions that use some kind of watchdog service are a candidate that you must disable for installing an Exchange Server CU.

The following two PowerShell examples help you to prepare the Windows services for installing an Exchange Server CU.

Prepare for CU installation

In preparation for the installation of an Exchange Server cumulative update, you can use the following PowerShell commands.

# Disable and stop services or just stop services
# Add other services as needed

# Set SMEX service to manual and stop services
Get-Service -Name 'ScanMail*' | Set-Service -StartupType Manual
Get-Service -Name 'ScanMail*' | ?{$_.Status -eq 'Running'} | Stop-Service -Force

# Stop SMEX SQL Express instance
Get-Service -Name 'MSSQL*' | ?{$_.Status -eq 'Running'} | Stop-Service -Force

# Disable and stop ENow monitoring services
Get-Service 'ENow*' | Set-Service -StartupType Disabled
Get-Service 'ENow*' | ?{$_.Status -eq 'Running'} | Stop-Service -Force

# Stop NetBackup service
Get-Service -Name 'NetBackup*' | ?{$_.Status -eq 'Running'} | Stop-Service -Force

Post CU installation

After installing the Exchange Server cumulative update you should restart your computer. I recommend initiating a check for additional Windows Updates for the CU. This helps to ensure that you do not only have the latest CU installed, but required security updates as well.

# Enabling and starting services
# Adjust the list of services as needed

# Enable and start SMEX services
Get-Service -Name 'ScanMail*' | Set-Service -StartupType Automatic
Get-Service -Name 'ScanMail*' | Start-Service

# Enable and start ENow Monitoring services
Get-Service -Name 'ENow*' | Set-Service -StartupType Automatic
Get-Service -Name 'ENow*' | Start-Service

Enjoy Exchange Server.

This year's MCT Summit Middle East took place on 19-20 March, with two full days of content about Microsoft technologies. Four session tracks plus a workshop track offered possibilities to learn and share knowledge, not only with the MCT community. 

The session recording will be made available on the event website.

I enjoyed talking about Exchange Hybrid, what it is, why you need it and how you implement it using the Hybrid Configuration Wizard.

My PowerPoint presentation is available on Slideshare. 

I look forward to next years' event. 

Links

• MCT Summit Middle East
• My PowerPoint slide deck on SlideShare 

Wouldn't it be nice if you could stay in Microsoft Teams even when working with your emails? Microsoft Teams is getting even more mature by providing an App for integrating Outlook into the Microsoft Teams client.

At least it seems that this is the case. or was it?

The Outlook App (Preview)

Last week a new App, still in preview, showed up in the Teams Admin Center (TAC) of some demo tenants. I filtered the list of available apps using Outlook as a search term.

When selecting the app, the details showed that the app was still in preview (v0.41).

As long as the app is enabled within Teams, you can add the app to a Teams Setup Policy and make it available for users.

Your users can then access the Outlook App using the Teams app bar.

You can simply click on Outlook in the app bar and, voilá, you have your Outlook inbox in Microsoft Teams. Due to the architecture of Microsoft Teams, you have access to your Outlook on the Web version of your inbox.

Links

• Learn more about Microsoft Teams 
• Teams App Setup Policy
• Learn more about today

Enjoy Microsoft Teams and Microsoft Outlook. - And remember today's date.

Being in a situation where you need to recover IT components or even a whole IT infrastructure after a disastrous event can be stressful. It would help if you stayed concentrated while dealing with the constant status requests by the management.

In such a situation, it can help to isolate your mind by listening to music. Music that helps you focus your thoughts, anger, and feelings in such a moment.

Maybe you find my Spotify playlist helpful. The playlist contains about 12 hours of music.

• IT Recovery Playlist

Listening to music using a headphone is good, but playing music loud and feeling the rhythm and every single bass is much better.

Enjoy.

I am honored to speak at Teams Nation 2021 on 12th May about Microsoft Teams Admin Roles and Teams Admin Center. 

Register today and visit my session in room Teams Nation Germany (TCD). As suspected, the session will be in German.

Teams Nation is a community conference and offers 115 sessions in 11 tracks.

Visit the conference site and register today.

• Teams Nation Conference
• Teams Nation Schedule
• Register for Teams Nation Conference

The use of Microsoft Viva requires using a modern SharePoint site as a home site. 

The documentation for enabling Microsoft Viva describes how to set a new SharePoint Online Home Site, but lacks an important step.

You must swap the new home site and current home site, in addition to setting the SharePoint Online home site. The swap cmdlet archives the current home site. 

# Replace with the new home site URL
$NewHomeSiteUrl = 'https://varunagroup.sharepoint.com/sites/Varuna'

# Replace with current home site URL
$CurrentHomeSiteUrl = 'https://varunagroup.sharepoint.com/'

# Replace with an archive URL
$ArchiveUrl = 'https://varunagroup.sharepoint.com/sites/oldhomepage'

Invoke-SPOSiteSwap -SourceUrl $NewHomeSiteUrl -TargetUrl  -ArchiveUrl $ArchiveUrl

Links

• Install Viva Connections today

I am honored to speak at CloudEight Virtual Summit 2021.

CloudEight is a conference about Microsoft 365 related topics and takes place for the third time. 

My session covers the topic "Microsoft Teams and On-Premises Mailbox - Why?"
What are the reasons for using Microsoft Teams with on-premises mailboxes, what are the technical requirements, and why is this approach not necessarily a good idea?

Register today.

Looking forward to seeing you at the virtual conference.

Links

• CloudEight Virtual Summit Event Homepage

Recently I had to support the uninstall of Exchange Server 2016 CU10 on a Windows Server 2019 system. That this setup is not supported is a different topic. In this case, a new Exchange Server 2016 system was placed in service, and the old system needed to be removed from the on-premises Exchange organization.

We mounted the Exchange 2016 CU10 ISO, and ran the following command from an administrative command line:

Setup.exe /mode:uninstall

Prerequisites Checks

The prerequisites check failed with an odd error:

http://terenceluk.blogspot.com/2017/01/attempting-to-delete-exchange-server.html

Querying for any incompleted public folder migration requests returned no results. But the prerequisites check insisted that there was an existing public folder migration request. In such a case you already know that you have to use ADSIEdit to find the object in question. 

It turned out that the prerequisites check was right, as we found a single public folder migration request in the Active Directory configuration partition. The request was an artifact of an unsuccessful migration attempt in 2019. After we have checked that the current modern public folder hierarchy worked as expected, we deleted the artifact from Active Directory.

Now the uninstall procedure passed the prerequisites check successfully and the uninstaller moved on removed Exchange Server 2016 step by step.

Until...

Uninstall Error

The uninstall step Language Files an Access Denied exception while executing MSIEXEC uninstall actions for each Language Pack.

Language Files                                                                                    FAILED

The following error was generated when "$error.Clear();
$regPath='HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall';
$PackageGUIDRegEx = "{DEDFFB[0-9a-fA-F]{2}-42EC-4E26-[0-9a-fA-F]{4}-430E86DF378C}";
$InstallPath = (Get-ItemProperty 'HKLM:\SOFTWARE\Microsoft\ExchangeServer\v15\setup').MsiInstallPath;

if(test-path ($regPath))
{
Write-ExchangeSetupLog -info ("Removing " +  $RoleLanguagePackType + " Language Packs.");
Get-ChildItem ($regPath) | foreach{
if($_ -match "(?<ProductCode>$PackageGUIDRegEx)") {
$langPackPackageCode = $matches['ProductCode'];
if($langPackPackageCode -ne $null -and $langPackPackageCode.Length -ne 0) {
Write-ExchangeSetupLog -info ("Removing package $langPackPackageCode");
$language = $langPackPackageCode.Substring(20,4);
$logFilePath = [IO.Path]::Combine($RoleLogFilePath,"Uninstall") + '.' + $language +
'.' + "Client" + "." + $RoleLogDateTime + ".msilog";
uninstall-MsiPackage -ProductCode ($langPackPackageCode) -LogFile ($logFilePath);
};
};
};
Get-Childitem -Path $InstallPath -include ".Localized.js",".Localized.min.js" -recurse | foreach ($) {remove-item $.fullname};
Write-ExchangeSetupLog -info "Remove Language Packs completed.";
};
" was run: "**System.UnauthorizedAccessException: Access is denied** ---> System.ComponentModel.Win32Exception: Access is denied
--- End of inner exception stack trace ---
at System.Management.Automation.Utils.NativeDirectoryExists(String path)
at System.Management.Automation.SessionStateInternal.IsItemContainer(CmdletProvider providerInstance, String path, CmdletProviderContext context)".

Interestingly, the ExchangeSetup log file showed that the uninstaller wrote the informational text Remove Language Packs completed successfully. 

Solution

After following an idea to remove the language pack-related registry keys and other fancy approaches, we did something trivial. We restarted the server, mounted the ISO file, and ran Setup.exe /mode:uninstall again. 

The uninstaller process now passed the step Language Files without any issues.

I sometimes like simple solutions.

Enjoy Exchange Server. 

When you create or update an Exchange hybrid configuration using the Hybrid Configuration Wizard magic things happen. That's why it is called a Wizard.

One essential step of the Hybrid Configuration Wizard (HCW) is the configuration of the hybrid mail-flow. The hybrid mail-flow is required for both, classic and modern Exchange hybrid. 

The wizard asks you to select one or more Exchange servers that you will utilize for handling inbound mail traffic from Exchange Online to your on-premises organization. You either configure direct mail flow to your Exchange Mailbox Servers in your internal company network, or to your Edge Transport Servers located in the perimeter network.

The following screenshot example shows the selection dialogue.

You can only select a server object, but not a receive connector on that selected server. The HCW chooses the "right" receive connector on the selected servers for you. If you are using the default set of receive connectors, you will not encounter any issues. HCW will use the default frontend connector on a mailbox server. When you use an Edge Transport Server you will run into any trouble as well. There is only one receive connector which you must extend by setting some additional parameters.

But what about an Exchange Organization where each mailbox server hosts multiple receive connectors bound to TCP port 25? 

The Problem

When you use multiple receive connectors bound to TCP 25 you will see that HCW will choose a receive connector that you won't expect. You might think that HCW will select always the default frontend connector. That is not the case. 

When you select multiple servers for hybrid mail-flow, and each server has a different receive connector configuration, you might get the impression that HCW selects the receive connector randomly. That is not the case either.

While doing some testing in a large enterprise infrastructure with five different Exchange forests (development, testing, staging, pre-production, production) we saw an interesting behavior.

From all available receive connectors having a TCP 25 binding, HCW selects the receive connector with matching RemoteIPRanges values of:

• IPv6 all (::-ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff) and IPv4 all (0.0.0.0-255.255.255.255)
  This is normally the default frontend receive connector when you do not adjust the RemoteIPRanges parameter
• Just IPv4 all (0.0.0.0-255.255.255.255)
• Just IPv6 all (::-ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff)
• IPv6 any address and IPv4 any address
• Just an IPv4 address

Adjusting the default receive connector does have a direct impact on how HCW selects a receive connector in your Exchange environment. When you use multiple receive connectors for internal relay purposes, your receive connectors might end up in a messing situation. As mentioned, HCW selects receive connectors with a TCP 25 binding, regardless of the transport location of the connectors, frontend, or hub transport. The enterprise environment mentioned had some deviations between the different environments and we saw TCP 25 receive connectors in frontend transport and hub transport. 

The Solution (sort of)

Run the HCW and select only one server for hybrid mail-flow and identify the receive connector configured by HCW. Configure an appropriate receive connector on all other mailbox servers used for hybrid mail flow. Update the hybrid configuration object of your on-premises Exchange Organization accordingly. 

Verify the following two Tls* parameters of the receive connector:

Get-ReceiveConnector 'EXSRV01\Default Frontend EXSRV01' | fl tls*
TlsCertificateName    : <I>CN=Sectigo RSA Domain Validation Secure Server CA, O=Sectigo Limited, L=Salford, S=Greater
                        Manchester, C=GB<S>CN=mail.varunagroup.de, OU=PositiveSSL, OU=Domain Control
                        Validated
TlsDomainCapabilities : {mail.protection.outlook.com:AcceptCloudServicesMail}

You must ensure that the hybrid receive connector uses the correct TLS certificate, enabled for SMTP. Additionally, you must set the TlsDomainCapabilitiers to allow cloud mail for connections incoming connections with a TLS certificate for mail.protection.outlook.com.

Keep your receive connectors at frontend transport.   

Links

• Exchange Server Mail Flow and the Transport Pipeline
• Exchange Hybrid - What, Why, and How - Presentation @ Slideshare

Enjoy Exchange Server.

The Exchange Product Group announced Exchange Server vNEXT for fall 2021. angekündigt. We are all very excited to see what the new version has to offer.

But what is the current situation in on-premises Exchange organizations? I have put a short questionnaire online for gathering information from you. 

The questionnaire deals with the currently used product versions of Exchange Server, the size of your Exchange organization in terms of the number of servers and mailboxes, and the use of planning for a hybrid configuration with Exchange Online.

Take the questionnaire following this link: https://forms.office.com/r/d9syBcgkMk

Thank you for your participation.

Viel Spaß mit Exchange Server.

The Microsoft 365 Collaboration BootCamp takes place on 21th & 21st  August 2021.

The event addresses collaboration and best practices for using Microsoft Teams, SharePoint, Lists, Groups, and Microsoft Security & Governance.

I am honored to speak about one of my favorite Topics: Microsoft Teams and On-Premises Mailboxes - Troubleshooting 101

Join my session on Saturday 21st August at 12:00 pm (GMT/UTC)

• Learn more about the conference here: https://events.powercommunity.com/microsoft-365-bootcamp/
• Register today to secure your seat.

The Problem

You might face a situation during an Exchange Server migration where your Exchange Server 2019 mailbox users are not able to open their public folder favorites when using Outlook on the Web (OWA).

When your users try to access a public folder, they receive an error message.

This error occurs when the public folder mailboxes are still hosted on a previous version of Exchange Server. This includes Exchange Server 2016 and 2013.

The online documentation explains, why this is happening:

• Access public folders located on servers running previous versions of Exchange

The Solution

The solution to this problem is easy. Move the public folder mailboxes to Exchange Server 2019 before you migrate any user mailboxes. 

This approach ensures that mailboxes hosted on Exchange Server 2019 and previous versions of Exchange Server are able to access public folders using Outlook on the Web.

Links

• Exchange Server 2019 - FAQ Client Limitations

Enjoy Exchange Server.

The Problem

Mail flow from on-premises devices and applications to Exchange Online is a tricky topic. The documentation allows for different solutions.

Recently a client ran into a situation where an on-premises application was not able to deliver messages to a configured inbound connector in the Exchange Online tenant. The connector was configured for remote IP address selection.

Exchange Online responded to each connection attempt with the following error message:

• 451 4.4.3 Temporary server error. Please try again later ATTR3.1

There weren't any changes on the on-premises configuration and the setup was in use for multiple months without any issues.

The Solution

It took some time to identify the solution, but in the end, the solution was easy.

Disabling and re-enabling solved the issue.  

Enjoy Exchange Online.

You might see the following error in the Windows Application Event Log:

• Source: MSExchangeApplicationLogic
• Event ID: 3018
• Level: Error

The request failed. Mailbox:  
Url: https://officeclient.microsoft.com/config16?CV=15.1.2308.14&Client=WAC_Outlook&corr=910a665e-fd34-4c4b-8ed9-8a38fb515483 

Exception: 
System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. 
---> System.IO.IOException: Unable to read data from the transport connection: 
   An existing connection was forcibly closed by the remote host. 
---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
   at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
   --- End of inner exception stack trace ---
   at System.Net.TlsStream.EndWrite(IAsyncResult asyncResult)
   at System.Net.ConnectStream.WriteHeadersCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at Microsoft.Exchange.Data.ApplicationLogic.Extension.BaseAsyncOmexCommand.<>c__DisplayClass18_0.<EndGetResponseCallback>b__0()

The request is successful when you try to connect to the URL provided in the error details using a browser on the Exchange server. 

Reason

You can verify that the issue by trying to access the URL using the PowerShell Invoke-WebRequest cmdlet. Open a new PowerShell session and try connecting to the URL.

$uri='https://officeclient.microsoft.com/config16?CV=15.1.2308.14&Client=WAC_Outlook&corr=910a665e-fd34-4c4b-8ed
9-8a38fb515483'

Invoke-WebRequest -Uri $uri

You will receive the same error message as stated in the event log by MSExchangeApplicationLogic. A successful connection returns XML as content.

The reason for this error is related to the .NET Framework TLS configuration, not Exchange Server. The .NET Framework lacks configuration for the use of TLS 1.2.

Solution

The solution for this issue is to configure the .NET Framework to correctly use TLS 1.2. You can follow the description for TLS 1.2 enforcement for Azure AD Connect, or you can simply use this Gist. 

Due to the changes made to the SCHANNEL configuration you just restart the computer to bring the changes into effect.

Note

Changing the TLS settings does not only affect outgoing connections but incoming connections as well.

Test the TLS changes in a test environment before adjusting your servers in the production environment. If you have not already enabled TLS 1.2 for your Exchange Servers, I recommend reading the 3-part series by the Exchange product group.

Links

• Gist - Set-TLS12.ps1
• Exchange Server TLS guidance, part 1: Getting Ready for TLS 1.2
• Exchange Server TLS guidance Part 2: Enabling TLS 1.2 and Identifying Clients Not Using It 
• Exchange Server TLS guidance Part 3: Turning Off TLS 1.0/1.1

Enjoy Exchange Server! 

These are the results of the  Exchange Server Questionnaire from August 2021.

First of all, I want to thank all of you who participated in the questionnaire. The results are pretty interesting. Even though, that the results are not 100% representative they provide a high-level view of the Exchange Organizations, the mail flow configurations, and the future plans regarding hybrid and Exchange Online.

With 55 replies the questionnaire is far from being a comprehensive representation of the Exchange organizations. But the answers provide an idea of the Exchange landscape used by organizations globally.

1. Exchange Server Versions in use (Production)

Exchange Server 2016 is the dominant version currently in use, followed by Exchange Server 2019. The vast majority of 93% runs modern Exchange Server versions. But there are still older and unsupported Exchange Server versions in use. 7% use Exchange Server 2010 and older. 

2. How many Exchange Server systems do you operate?

76% of the organizations maintain up to ten Exchange servers. 20% prefer to rely on just one Exchange server. It is interesting that only 2 (not percent) plan to go hybrid or to move to Exchange Online.  

3. How many mailboxes do your Exchange Servers host?

The majority of on-premises Exchange organizations are in the 1,000 - 10,000 mailboxes range. Nevertheless, the SMBs with 1 to 1,000 mailboxes adds up to 50% of the Exchange organizations that took part in this questionnaire. There are just a few organizations that host more than 50,000 mailboxes.    

4. Do you use an on-premises or cloud-based SMTP gateway solution?

There are Exchange organizations that do not use an SMTP-Gateway solution as part of the mail-flow implementation. Thor organizations that do not use a gateway solution run 1 to 10 Exchange servers on-premises. The majority of those have less than 1,000 mailboxes but there are a few that are responsible for more than 1,000 mailboxes. That leaves the question of why an organization prefers to not secure mal-flow with a gateway.

5. Which product do you use as a gateway solution?

The use of SMTP gateways is a must, as you do not want to expose your domain member servers to the Internet, not even for the SMTP protocol. A majority of 28 answers for other gateways shows, that there are so many products available and that I did not choose valid answer options upfront. 

The Other answers include:

• Cisco ESA
• Clearswift
• Eleven
• Fortigate
• IronPort
• Postfix
• Reddoxx
• Trustwave

6. Is your current Exchange organization using a hybrid configuration with Exchange Online?

65% of the Exchange organizations of this questionnaire already run in a hybrid configuration with Exchange Online. Only 35% are (still) not using a hybrid setup.  

7. Do you plan to implement a hybrid Exchange configuration or to move to Exchange Online?

Of those who currently do not run a hybrid configuration only 37% plan on implementing Exchange Hybrid or migrate fully to Exchange Online. Staying on-premises is the only option.

8. Until when do you plan to implement a hybrid configuration or go cloud-only?

The majority of the organizations still running only an on-premises Exchange organization plan on implementing Exchange Hybrid or migrating to Exchange Online by the end of 2021. None of the participating organizations has plans scheduled after 2022.

9. Which hybrid model did you choose?

It is no surprise that Classic Full Hybrid is the most adopted hybrid configuration. And, no surprise either, none of the other classic hybrid options is implemented. The modern hybrid approach is implemented but with lesser.

10. What are the reasons for staying 100% on-premises?

The reasons for staying with an on-premises Exchange organization vary. the reasons mentioned are:

• Enclosed environment, external access with BlackBerry UEM, due to public sector data security requirements
• Mailbox data is classified as too sensitive
• Too expensive and low internet bandwidth
• Legal and clients audits 

There are still organizations that choose an on-premises Exchange organization in favor of Exchange Online. I wonder if company policies for reducing the carbon footprint might drive the migration of on-premises data center resources to hosted cloud services.  

11. Will you implement Exchange Server vNEXT?

Exchange Server vNEXT is in scope for 47% of the organizations. When comparing it with the used Exchange Server version currently in use (~50% Exchange Server 2016) it is an indicator that some companies just skip Exchange Server 2019. Some organizations prefer not to follow the full life-cycle of Exchange Server. s7% of those who do not want to implement Exchange Server vNEXT and want to stay on-premises are single server implementations of Exchange. 

Summary

The product Exchange Server is still widely used in on-premises deployments. The reasons vary from legal and compliance requirements, network bandwidth constraints, and the overall costs for Exchange Online. Exchange Server vNEXT is a must-have for nearly 50% of the organizations participating in this questionnaire. There are still older and unsupported versions in productive use. Why this is the case is unanswered in this questionnaire.

Organizations running a hybrid Exchange configuration primarily use a Classic Full Hybrid configuration. This might be due to an early implementation in those days when nothing else was available, or due to requirements using Microsoft Teams with on-premises mailboxes. The adoption of Modern Hybrid shows that the Hybrid Agent approach helps organizations that cannot implement a Classic Full Hybrid. 

I leave the results of this questionnaire to your interpretation and look forward to your replies, either to this blog post or by social media on Twitter and LinkedIn. Please use the hashtag #ExchangeQuest2021.

There will be a new Exchange Server questionnaire in early 2022, covering various implementation scenarios in more detail. If you want to see a specific Exchange topic covered in the 2022 questionnaire, just let me know.

Again, thank you all for participating in this questionnaire.

When you move mailboxes using migration batches you might encounter a situation that your batch contains migration users that fail during batch execution. One of the possible reasons is an existing move request for the affected users. You must remove those requests to successfully move mailboxes.

The following PowerShell example gets all failed migration users from a migration batch and removes existing move requests. 

$r = Get-MigrationUser -BatchId MyMugrationBatch | ?{$_.status -eq 'Failed'}
$r | %{Remove-MoveRequest -Identity $_.MailboxIdentifier -Confirm:$false}

Enjoy Exchange Server!

When you prepare your on-premises public folder hierarchy ACLs for migration to Exchange Online or for moving from Exchange Server 2016 to 2019 you might see the following error:

Multiple objects with legacy DN ADCDisabledMail were found.

This error prevents you from removing orphaned entries from public folder ACLs. And when you do not clean up the ACLs, you cannot migrate public folders to Exchange Online or move public folder mailboxes from Exchange Server 2016 to Exchange Server 2019.

The affected objects are mail-disabled objects that were disabled with Exchange Server 2010 or older. The older Exchange Server version used something called Active Directory Connector (aka ADC). When mail-disabling a user or security group, ADC stamped the legacyExchangeDN attribute with ADCDisabledMail. Modern Exchange Server versions do not write that value to the attribute when you mail-disabled the object.

To successfully migrate or move your public folders you must clear the legacyExchangeDN attribute. Otherwise, you cannot remove the orphaned ACL entries.

Simply use the following PowerShell script to clean up those objects.

PowerShell Script

Enjoy Exchange Server.

You are hopefully familiar with the new Exchange Emergency Mitigation Service (EEMS) for Exchange Server 2016 and 2019. That is a new service providing automated emergency configuration of your Exchange servers by Microsoft in the case a security risk has been identified. Such emergency mitigation is a technical workaround until a proper security patch is available.

The service responsible for fetching the current list of published mitigations is MSExchangeMitigation. 

Exchange Organisation following the official guidance for deploying Exchange Server won't see any specific issues with EEMS. It simply works. 

But Exchange Server runs in many different infrastructures where you might end up in a situation with a non-working EEMS.

Findings

EventID 1008 - MSExchangeMitigation service does not start

You see the following event log error:

Exception encountered while fetching mitigations : 
System.AggregateException: One or more errors occurred. 
---> System.Net.Http.HttpRequestException: An error occurred while sending the request. 
---> System.Net.WebException: The underlying connection was closed: 
      Could not establish trust relationship for the SSL/TLS secure channel. 
---> System.Security.Authentication.AuthenticationException: 
      The remote certificate is invalid according to the validation procedure.

In addition, you see the following in the diagnostic logs of the Exchange Server:

S:LogLevel=Information;S:Message=Started MSExchangeMitigation
S:LogLevel=Information;S:Message=Fetching mitigations from https://officeclient.microsoft.com/getexchangemitigations
S:LogLevel=Information;S:Message=Using Proxy http://[IPADDRESS]/ To Fetch Configurations
S:LogLevel=Information;S:Message=No diagnostic data sent. DataCollectionEnabled is false
S:LogLevel=Warning;S:Message=TLS certificate or its chain validation failed
S:LogLevel=Error;S:Message=Exception encountered while fetching mitigations :   One or more errors occurred.;S:Source=Microsoft.Exchange.Mitigation.Service.Mitigations.MitigationEngine

File location: V15\Logging\MitigationService

But what is the validation procedure failing? The solution is simple. The certificate revocation check for the certificate chain failed. The EEMS was not able to connect to the CRL-endpoints of each certificate in the certificate chain. CRL-endpoints are accessible by HTTP and not HTTPS for performance reasons. And outbound HTTP is often blocked for Exchange servers. 

The Exchange Server must be able to validate the certificate chain successfully establish a TLS-connection to officeclient.microsoft.com. Certainly, you can disable the CRL check for the server. But this is something I do not recommend. The XML file containing the mitigation configuration is signed by an X509 certificate and your servers should be able to validate and check the CRL. 

Solution

Ensure that your Exchange servers can communicate with the Internet to validate the certificate chain.

Links

• New security feature in September 2021 Cumulative Update for Exchange Server

Enjoy Exchange Server.