Project Honolulu and Exchange servers

Recently Microsoft announced a new web-based server management console, it’s called Project Honolulu. Looking at the documentation and screenshots this is going to be a huge step forward from the native sever management consoles that we have seen from Server 2008-2016.

A technical preview of Project Honolulu is already available for download: https://aka.ms/HonoluluDownload Project Honolulu can be used to manage servers with Windows Server 2012, 2012 R2 and Server 2016. However, there is a requirement to install PowerShell v5 on those older servers. From the documentation:

Honolulu requires PowerShell features that are not included in Windows Server 2012 and 2012 R2. If you will manage Windows Server 2012 or 2012 R2 with Honolulu, you will need to install Windows Management Framework (WMF) version 5.0 or higher on those servers.

Unfortunately Microsoft does not support installing newer versions of WMF on servers that have Exchange installed. The Exchange Supportability Matrix says the following about the supported versions of WMF on servers running Exchange:

Version of Windows Management Framework built into the release of Windows Server you're installing Exchange on.

This means that installing PowerShell 5 on Server 2012 or Server 2012 R2 is not supported for servers running Exchange, effectively making it impossible to use Project Honolulu to manage Exchange servers. The only exception is Exchange 2016 running on Server 2016 because this OS was released with PowerShell v5 natively and meets the Project Honolulu requirements.

Exchange Remote PowerShell behind a proxy server

Today I ran into the following issue. I tried to open a Remote PowerShell session to Exchange Online but the module failed to load with the error message:

New-ExoPSSession : Create Powershell Session is failed using OAuth

Initially I suspected that the customer’s firewall performed some kind of traffic inspection or recognized and denied the traffic as a blocked ‘application’. I was able to browse the internet from this computer and after all Remote PowerShell is just https traffic.

However, the culprit was the proxy server. More specific, I configured a proxy server in the Internet Explorer settings but did not verify if Remote PowerShell observed the same connection settings. And the answer is no, the console does not use the IE settings but falls back to the winhttp configuration. In this environment the firewall configuration did not allow outbound traffic unless it passes through the proxy.

Telling PowerShell to use the proxy server is easy. Open an elevated prompt and use the following command to configure winhttp to use a proxy server and port:

netsh winhttp set proxy proxy.network.tld:8080

To import the settings that where configured in IE:

netsh winhttp import proxy source =ie

And important, to remove the proxy configuration for winhttp:

netsh winhttp reset proxy

Configuring the proxy for winhttp fixed my issue and I was able to connect to Exchange Online with PowerShell successfully.

Some feedback on the new Exchange PowerShell cmdlets

As someone who has been working pretty much dedicated with Exchange since ~2000 I have witnessed quite a few changes. The single most important one is the move to PowerShell as the primary management interface. The Exchange team was the first within Microsoft to switch for the full 100% to PowerShell and at least two version of Exchange have seen their release slightly delayed because they had to wait for the PowerShell version GA first.

While all other products continued to implement PowerShell as well and used the same set of professional standards there were significant differences to notice between the various implementations. I always preferred the way how the Exchange team implemented PowerShell. Identity is always positional parameter 1, Get cmdlets without a parameter return the first 1k of results or whatever is the default value of ResultSize for that cmdlet and destructive cmdlets (Set, Remove, New) always support the -WhatIf switch. I’m sure that every Exchange admin once tried to run Get-ADuser and discover that the cmdlet doesn’t return any values unless you specify a value for -Filter.

Personally I always preferred the consistent and user friendly implementation in Exchange. Maybe that’s why it bothers me that Microsoft no longer applies the same standard to new cmdlets that are being added in Exchange and Exchange Online.

An example is Get-FoucesedInbox and Set-FoucesedInbox:

The first example shows that the -Identity parameter no longer is a the first positional parameter, a positional parameter can be omitted and PowerShell assumes the first value after the cmdlet to be meant for the first positional parameter. This can be a bit annoying because Exchange admins are no longer used to specify the -Identity parameter because they never had to.

What’s more serious is that Set-FoucesedInbox no longer accepts the -WhatIf switch. -WhatIf tells the cmdlet to perform all prerequisite checks but not to make the actual changes. This parameter is essential for admins to check their syntax before running the actual cmdlet.

Some other examples of cmdlets that did not receive the -WhatIf switch are:

Cmdlet	Exchange	Exchange Online
Set-Clutter	X	X
Set-CompliancePolicySyncTenantInfo	X	X
Remove-BlockedSenderAddress		X
Remove-CompliancePolicySyncNotification	X	X
Set-OMEConfiguration		X
New-ReportSchedule		X
Set-ReportSchedule		X

As you may notice they were all added in the last few years.

While I understand that the industry has changed and Microsoft’s priorities are different in the ‘cloud first’ era, I do urge the Exchange Team to keep focus on what made them successful in the first place: excellent quality and always strive to deliver the best. Please do not forget to think of the people that are actually working with the product.

Let’s not run Exchange 2016 Edge Transport on Windows Server 2016…

Another example where old technology meets new technology. Old is in this case the Content Filtering agent on an Edge Transport server, new being Windows Server 2016. In short, the SmartScreen technology that is used in the Content Filtering agent conflicts with SmartScreen implementation in Windows Server 2016 operating system.

At this point this is known to prevent a proper uninstall of Exchange and can cause crashing transport services, this applies to an Exchange 2016 Mailbox server where the admin installed the anti-spam agents too.

From today Microsoft no longer recommends to install the Exchange 2016 Edge Transport role on Windows Server 2016, the same goes for installing the anti-spam agents on the Mailbox role when installed on Windows Server 2016.

What do you need to know about Exchange 2016 on Windows Server 2016:

• Requires Exchange 2016 CU3 or newer
• IIS bug causing issues with DAG members, fixed with KB3206632 and now a prereq for CU4
• Edge Transport role not recommended
• Anti-spam agents on Mailbox role not recommended

An interesting note is that the author of the post on the Exchange Team Blog consistently uses the term ‘Edge role’ instead of Edge Transport. This could be one more change in the branding but more likely just a mistake.

New Exchange Online PowerShell module with MFA support no longer in Preview

In a previous article I wrote about the new PowerShell module with support for Modern Authentication and Multi-factor Authentication. The Preview status was a reason for many organizations to hesitate to take the new module into production.

Yesterday Microsoft released an updated version of the Hybrid Configuration Wizard with MFA support, the HCW now requires installation of the new PS module to support MFA. Microsoft’s ‘hybrid’ PM Timothy Heeney confirmed in the comments section that this also marks the official RTM of the new PowerShell module, the module is no longer in Preview. Good news!

Surface Hub devices and the Skype for Business Trust Model

I’m sure that most Lync or Skype for Business admins, users as well, are familiar with the Trust Model. The Trust Model is responsible for the ‘Skype for Business cannot verify that the server is trusted for your sign-in address’ warning. This warning is thrown when the clients tries to create a secure TLS connection with a server and the domain suffix of the server is different from the user’s SIP address, the server can be either a Skype for Business or Exchange server. This warning is very common in organizations that use more than one SIP domain and is often suppressed on managed computers with the TrustModelData registry value.

With a Surface Hub device the issue is a bit more complicated to determine, but very easy to work around. In this article I will explain how.

Let’s consider the following scenario. Contoso is an enterprise organization that uses many different SMTP and SIP domains across their divisions. The AD domain name is contoso.com and this is also the DNS domain suffix for most servers. Skype for Business is hosted with a 3rd party named Fabrikam, their servers have a fqdn with the fabrikam.com suffix.

The Northwind Traders division of Contoso has purchased a Microsoft Surface Hub device and created a device account with a SMTP, UPN and SIP address with a nwtraders.com suffix.

The issue

An admin was able to configure the Surface Hub with this computer account, however users are not able to start a meeting.

It’s important to understand that although the device boots successfully, the built-in Skype for Business client is not immediately connecting to Skype for Business (Online) but starting a meeting does trigger this process.

The investigation

Surface Hub devices run on Windows 10 Team edition which does not offer a regular interface that allows to access the file system to collect log files. Instead we need to boot the device, let it run for 5 minutes, then reproduce the issue and tell the Surface Hub to collect the log files.

To do this, connect a USB disk to the device and open the Settings app. Then navigate to Update and Security, Recovery, Collect logs. The log files are now written to the USB disk.

When analyzing the log files, be aware that the Surface Hub’s Skype for Business client is very similar to the Lync 2010 Windows Store app and behaves as a mobile client.

2757 TL_WARN() [2]10B0.1394::02/28/2017-12:38:54.103.00000a8c (NONE,NModel::CTrustModelManager::LookupTrustModel:CTrustModelManager_cpp124)<O_TRC><ADR>0x00000243A9F16EB0</ADR>Trust model for server rp.contoso.com not found. hr=0x80ee0058</O_TRC>
2758 TL_WARN() [2]10B0.1394::02/28/2017-12:38:54.103.00000a8d (NONE,NModel::CTrustModelManager::QueryTrustModel:CTrustModelManager_cpp171)<O_TRC><ADR>0x00000243A9F16EB0</ADR>Server: rp.contoso.com cert=0000000000000000, blockAndWait=0</O_TRC>
2759 TL_INFO() [2]10B0.1394::02/28/2017-12:38:54.103.00000a8e (NONE,NModel::CTrustModelManager::QueryTrustModel:CTrustModelManager_cpp230)<O_TRC><ADR>0x00000243A9F16EB0</ADR>Not able to get SAN from cert. Continue query TrustModel.</O_TRC>

Here we clearly see the issue. The DNS domain suffix of the reverse proxy server is contoso.com and the user’s SIP address suffix is nwtraders.com. This triggers the Trust Model warning and because the Surface Hub interface does not present the familiar warning, it simply prevents the device from connecting with Skype for Business.

The solution

As I mentioned earlier, this is a very common issue for most organizations. The Surface Hub device offers an interface to add domains to the Trusted Domain list. Open the Settings app and navigate to This device, Calling. Here click the Configure domain name and enter a comma separated list of the additional domain names that exist on your Skype for Business and Exchange servers.

In this scenario we would need to enter the DNS suffix of the reverse proxy, but that’s not sufficient. While this will allow us to connect to the reverse proxy this will throw another warning in the logs because the DNS suffix of the front-end server is different from the user’s SIP address suffix too. In this example we would need to enter the following:

contoso.com, fabrikam.com

A reboot of the device is required to activate the new settings. If you’re still not able to connect, export and analyze the logs again. There may be additional issues that prevent the device from connecting to Skype for Business.

Summary

Instead of showing a warning popup the Surface Hub simply does not allow to connect when the domain name of a servers is different from the SIP domain. If you know that this scenario applies in your organizations, add the additional domains in the Settings app.

For more information please see:

• Lync cannot verify that the server is trusted for your sign-in address" message during client sign-in
• Surface Hub administrator guide: Configure domain name for Skype for Business

Skype for Business PSTN Calling in Preview for The Netherlands and Ireland

PSTN calling is an add-on telephone service that, when combined with Skype for Business Cloud PBX, can become your phone system. This is currently available for users in the United Kingdom, United States, Puerto Rico, France and Spain.

Microsoft now introduced this feature in Preview for The Netherlands and Ireland. To nominate your organization, make sure you’re prepared to deploy PSTN Calling to at least 50 users and are available to give feedback on the experience.

Based on what we’ve seen with France and Spain I expect general availability in two or three months. More information on https://www.skypepreview.com/

New 100 GB mailbox limit currently being rolled-out to your tenant

In early December (2016) Microsoft announced that the default mailbox limits in Exchange Online would be raised to 100 GB, a massive increase from the 5 GB that we considered to be huge in the BPOS era. Who needs such a large mailbox? Well, we do now apparently.

While discussing this with my peers last week we noticed that most of us did not saw the change reflected in their tenants. Today I checked it again and saw that the new limits were applied to my mailboxes.

Note that the limits only change for mailboxes where the default limits were not updated. If you have set one of the three values to a non-default value, no changes were made automatically. Admins are able to change the values manually up to 100 GB.

Another thing to be aware of is that the new limits only apply to Exchange Online licenses that are purchased as part of an E3 or E5 subscription. If you’re purchasing Exchange Online as a separate SKU the old limit of 50 GB remains unchanged, the same applies for shared and resource mailboxes. For more information, see the Mailbox storage limits section in the Exchange Online Limits document on TechNet.

Fix for the Skype for Business hangs

The latest update for Office 2016 in Current Channel contains a fix for Skype for Business issue where the interface stops responding when the user has multiple IM windows open.

The build number is 7668.2074, my Office 365 ProPlus was still on a slightly older build so I had to start the update process manually.

For more information about the latest updates for Current, Deferred and First Release for Deferred, see Office 365 client update channel releases.

A few notes on running Exchange in Microsoft Azure

Earlier this week Microsoft published a very interesting article: Exchange 2016 dev/test environment in Azure. I’m not a big fan of the how-to format of articles when the topic is around deployment because the guidance and specifics tend to change with later updates. There’s the risk of your article becoming obsolete sooner than planned.

The article contains an excellent real-world example of how to deploy a few virtual machines on Azure. I like how the author used PowerShell to configure the VM TCP/IP configuration and Install-ADDSForest to promote the server to domain controller.

Cool, let’s do this in production!

Although the article clearly states that the goal is the deploy Exchange for test and development, this article may spark new interest for people who would like to run Exchange in production too. The good news is that running Exchange in Azure is supported, but similar to running Exchange on other virtualization platforms some additional requirements must be met. The most significant requirement is actually mentioned in the article, in the section where storage is assigned for the Exchange VM:

This information can be found in the article Exchange 2016 virtualization as well:

Deployment of Exchange 2016 on Infrastructure-as-a-Service (IaaS) providers is supported if all supportability requirements are met. In the case of providers who are provisioning virtual machines, these requirements include ensuring that the hypervisor being used for Exchange virtual machines is fully supported, and that the infrastructure to be utilized by Exchange meets the performance requirements that were determined during the sizing process. Deployment on Microsoft Azure virtual machines is supported if all storage volumes used for Exchange databases and database transaction logs (including transport databases) are configured for Azure Premium Storage.

Running a single VM with the proper resources for Exchange 2016 24/7 in Azure is probably more expensive than you would think, with prices starting around € 500,- per month for the VM, two small premium storage disks and a Windows license.

While this falls outside of the scope of the article we’re discussing here, I would like to mention the requirement to use a smart host for outbound email delivery if you choose to run Exchange in Azure.

Make sure you do the math and understand the requirements before using this article as an alternative deployment plan for your next Exchange server. At this moment I am not aware of any organization running Exchange servers in Azure for production. If you are doing this, please reply in the comments.

But wait, there’s more

As I mentioned in my introduction Exchange installation how-to articles often contain wrong or obsolete information. Unfortunately that applies to this article too.

The latest version of Exchange, that’s actually great advice. Unfortunately the url https://go.microsoft.com/fwlink/p/?LinkId=747753 links to the download page for Exchange 2016 CU1. Not only is this version much older than the latest version, currently CU4, more importantly this version of Exchange is not even supported to run on Windows Server 2016. It was Exchange 2016 CU3 that introduced support for Windows Server 2016, although CU4 is recommended because a compatibility issue in Server 2016.

Exchange 2016 not supported for virtualization?

And while making some screenshots for this article I found another gem. On the page with the requirements for virtualization is this weird segment:

This is actually a left-over from the Exchange 2007 era, where this modernized virtualization policy was introduced. The requirement on that page was that Exchange 2007 SP1 was required and list of supported guest operating systems indicated that running a VM with Windows Server 2003 x64 was not supported.

With every new edition (2010, 2013, 2016) this page was copied without much changes. Instead of remove this no longer relevant section, the writers updated the section whit what they thought made sense. That’s what a VM running Exchange requires that it’s running Exchange 2010.

But I digress, back to Exchange 2016 now. The current version of this page does not include Windows Server 2016 as a supported operating system for running an Exchange 2016 VM. While this is obviously a mistake, technically speaking a virtualized Exchange 2016 server installed on Windows Server 2016 is currently not supported.

I found a doc error too, what should I do?

Shoot an email to Ex2013HelpFeedback@microsoft.com and make sure to include the url of the page, a quote and/or screenshot of the text you’re referring to and an explanation of why you think it’s in error. The team behind this alias is awesome and almost every time you will receive a response from an actual person.

Windows Management Framework (WMF) 5.1 Released, do not install on Exchange

Today the PowerShell team announced the release of version 5.1 of the Windows Management Framework, for most people better known as PowerShell 5.1. While Windows Server 2016 already contained WMF 5.1 when the product was released, the download released today allows administrator to install WMF 5.1 on older operating systems such as Windows Server 2008 R2, 2012 and 2012 R2.

This is a reminder that currently no version of Exchange Server supports a newer version of WMF than the version that was released with the operating system. As always when it comes to supportability questions, the Exchange Server Supportability Matrix is a valuable resource.

Strictly speaking the same limitation applies to a remote management computer too, but I am not aware of any issues with running a newer version of Remote PowerShell against Exchange on an operating system with an older version of WMF. But be aware of the supportability limitations around WMF and PowerShell.

Thanks Niklas, for pointing out that Server 2016 was released with WMF 5.1, not 5.0 as I wrote initially.

Microsoft about to release new Skype for Business IP Phone firmware for Polycom VVX devices

LCS/OCS/Lync/Skype has a long history when it comes to management of IP phone firmware updates. OCS 2007 R2 introduced the Device Update Service and greatly simplified the process and the required infrastructure. In the Lync 2010 timeframe the 3rd party IP phone (3PIP) certification added support for non-Microsoft firmware such as Polycom UC Software.

In Lync Server 2010 for instance, an admin uses the Import-CsDeviceUpdate cmdlet to import a .cab file and approves the update in the LSCP. The client device will periodically query the Device Update Web service and download and install the new firmware.

The same principle applies to Skype for Business Online with Cloud PBX today. Technically speaking the title of this article is incorrect, Microsoft is not releasing the software but merely distributing the Polycom firmware through it’s infrastructure.

Enabling or disabling this feature is a tenant-wide setting and can be done by modifying the CsIPPhonePolicy. By default EnableDeviceUpdate is True.

For more information about this feature, read Jeff Schertz’s post about this topic: Device Updates with Skype for Business Online.

The major difference with Skype for Business on-premises is that customers cannot upload a firmware version. Microsoft has an internal process to certify 3rd party firmware updates. At the moment of writing the only version that has been approved is Polycom UCS for the VVX line of devices, version 5.4.1.17653. Specifically the following devices are supported:

• VVX 201
• VVX 3xx
• VVX 44xx
• VVX 5xx
• VVX 6xx

The issue is that the most recent version of Polycom UCS is currently 5.5.1. That doesn’t sound much newer than 5.4.1 but in reality Polycom has released many interim versions and each version added important new features and fixes. That’s why it’s good to know that Microsoft is currently qualifying UCS version 5.5.1.11344.

If you consider using Skype for Business Online device updates, be aware that the capabilities currently are extremely limited. The device update feature cannot be enabled for a specific set of users or devices and the other management features are limited as well:

(note: EnableBetterTogetherOverEthernet by default is False)

Most of the configuration items are pretty self-explanatory but if you want to know more, the help page of  Set-CsIPPhonePolicy is very informative.

Microsoft did not communicate a release date for 5.5.1.11344 but my sources say that it shouldn’t take long. For more information, read: New features in the firmware update for Polycom VVX IP phones.

Note

I’m working with Polycom and Microsoft to investigate an issue where Polycom VVX devices receive a ‘400 bad request’ when they query the Skype for Business Online Update Service. If you’re encountering the same, let me know in the comments.

New Exchange PowerShell module with Modern Authentication support now available in Office 365 Portal

Many organizations are enabling multi-factor authentication (MFA) for all their accounts, or a subset such as for instance user accounts with an admin role. Especially with how easy this is with Office 365, even with the basic MFA feature that’s included with the license. Unfortunately enabling MFA on an admin account breaks the ability to use PowerShell to administer Exchange Online, Skype for Business Online or SharePoint.

A few months ago a new version of the Exchange PowerShell module was ‘leaked’ to the internet. It was a click-to-run executable without any documentation, but it introduced support for Modern Authentication which is a requirement for MFA.

And while there’s still no public announcement on the various Microsoft Exchange or Office blogs, not even a mention in the Office 365 Roadmap, there have been some recent updates. For starters the new PowerShell console is now available for download in the Hybrid section of the Exchange Admin Center.

The second is that some documentation has been published. The process was pretty self-explanatory but some official guidance is always better. The short version is this: install the application, then use Connect-EXOPSSession to create a remote session.

Read more in this article on the TechNet Exchange Technical Library.