Do not convert synced mailboxes to shared in a hybrid environment

Recently I ran into an issue with shared mailboxes in Exchange Online. In the Office 365 Admin portal I created a custom view in the Active users section to track issues with our licensing process:


Some users were displayed here which where moved from on-premises as regular mailboxes and then converted to shared with the button in EAC. Which is odd because a license isn’t required for a shared mailbox.


At a first glance everything seemed okay, the mailbox moved to the Shared view in EAC. The only thing off was the fact that LicenseReconciliationNeeded was set to True for this user which is why it showed up as mailbox without a license in the portal.


Next I compared the mailbox properties as well as the on-premises AD attributes with another shared mailbox, that happened to be shared before it was moved to Exchange Online.


So for the shared mailbox that was converted before the move msExchRemoteRecipientType was set to 100 (Shared mailbox in Exchange Online) while the other still displayed a value of 4 (Migrated mailbox).



Next step was to involve Microsoft Premium Support. Their first recommendation was to change the value from 4 to 100 in the on-premises AD and have it synced back to Azure AD. This didn’t make sense to me and by that time I already discovered that I could easily reproduce the issue in the test environment by converting a moved mailbox with the button in EAC so it was not caused by an issue in our on-premises AD.

Next the engineer provided the following KB article and explained that the behavior I’m seeing is ‘by design’: Shared mailboxes are unexpectedly converted to user mailboxes after directory synchronization runs in an Exchange hybrid deployment.

The article describes a scenario where a mailbox has been moved to Exchange Online, is converted to Shared and with the next directory sync is converted back to regular. Interestingly that last part is not happening in our environments and I was not able to reproduce the described behavior, not with a highly customized FIM nor with a default AAD Connect sync engine.

Anyway, accoring to the article it is not supported to convert a regular mailbox for a synced user to the shared mailbox type. The technical explanation is that directory synchronization currently doesn’t support the syncing back of all attributes that change during the conversion.

Microsoft recommends to move the mailbox to on-premises, convert it to shared and then move it back to Exchange Online. For some customers this may present an acceptable work-around, for others it is cumbersome and requires additional planning and communication to end-users.

And the real question of course is why Microsoft allows their customers to convert a synced mailbox to shared when the underlying technical issue can result in the mailbox being deleted after the grace period ends. I left feedback through the support channel and on the Office 365 Uservoice site. Please vote here if you agree this should change: Support conversion to shared mailbox for synced users

My take on the Star Wars The force awakens story line

Just kidding, this is a test post with version 5.1.2 of Open Live Writer. In case you missed it, Open Live Writer is the open source successor of the popular blogging tool Windows Live Writer. WLW once was part of the Windows Live Essentials bundle of tools and has a large share of fans, even though development and improvements are absent since 2012.

Read more about OLW here: http://openlivewriter.org/

Exchange 2013 CU11 and Exchange 2007/2010 Update Rollups released

Microsoft today announced the release of the 11th Cumulative Update for Exchange 2013. That's not all, Exchange 2010 SP3 receives Update Rollup 12 and there's even one for Exchange 2007: UR 18. The latter is basically a DST update so don't expect any other issues to be fixed after you install this update.

Click the link to read more. One more thing, make sure you read this article on the Exchange Team Blog too: Exchange Management Shell and Mailbox Anchoring. This change in behavior may impact the way you manage Exchange with PowerShell. There's been some concern in the community about the benefits of this change, some people suggest that it introduces more problems than is solves. More about this subject on Reddit: Major change to how PowerShell works with Exchange

Now the wait is for Exchange 2016 CU1 which will be released with the next set of semi quarterly updates at the end of 2016Q1.

December 8, 2015, update for Outlook 2010 causes Outlook to start in safe mode

I noticed several people reporting weird issues with their Outlook 2010 clients. For instance the loss of signature and other settings. The cause seems to be installation of update KB3114409. This update adds a new feature:

Adds administrative support to prevent Outlook 2010 from booting into safe mode. Administrators set this function in some scenarios when they have add-ins that must be enabled.

Admins can use a registry key or GPO to prevent the users from starting Outlook in safe mode. The issue is, without those controls Outlook now starts in safe mode by default.

The fix is relatively easy, simply uninstall update KB3114409. Alternatively admins can use the aforementioned registry key to control Outlooks behavior. Some smart people on Reddit figured out the correct entries for both x86 and x64 systems: KB3114409 breaking Office 2010, forcing it to start in Safe Mode, which is the exact opposite of what's supposed to happen.

Microsoft pulled the update so I expect no new occurrences. Anyway, this is why we roll-out patches to a small test group before rolling them out to the entire enterprise, don't we?

What every Office 365 admin should know about supported Outlook versions

Working with customers that are beginning to consume Office 365 services I see a lot of confusion about whether their old Office version should work with Office 365. Especially when we’re talking Office 2010 or even 2007. What certainly not helps is that Microsoft restructured their Office 365 documentation in such a manner that it is hard to find all relevant information.

The general Office 365 for Business requirements states the following:

Make sure that your Office clients are compatible with Office 365. Office 365 works with any version of Office in mainstream support: the latest version of Office, Office 2013, and Office 2011 for Mac. Previous versions of Office clients, such as Office 2010 and Office 2007, may work with Office 365 with reduced functionality.

The requirements for Exchange Online are more restrictive and definitely clearer, although they are hidden in the Exchange 2016 system requirements page:

Exchange 2016 and Exchange Online support the following versions of Outlook:

• Outlook 2016
• Outlook 2013
• Outlook 2010 with KB2965295
• Outlook for Mac for Office 365
• Outlook for Mac 2011

Outlook clients earlier than Outlook 2010 are not supported. Email clients on Mac operating systems that require DAV, such as Entourage 2008 for Mac RTM and Entourage 2004, are not supported.

So your users are still on an older Outlook 2010 build without SP2 and the April 15, 2014 update? Not supported. Outlook 2007? Not supported. From experience I can tell that these requirements are not a joke, your users will not be able to connect to Exchange Online with Outlook 2010 RTM. Another customer saw Outlook 2007 crashing after moving the mailboxes to Exchange Online and was able to work-around this by removing the language pack.

So if we like it or not, consuming a cloud service equals losing control about what client software you’re using. It’s the cloud provider that dictates the supported versions and when they think you should upgrade. That brings me to the final point, that these requirements not static and change over time. So make sure you visit the sources regularly to see of there have been updates.

Install an older Exchange version after last server was removed

Consider a scenario where an Exchange environment was upgraded from let’s say Exchange 2007 to Exchange 2010 (I know, old versions). Some time after the admin removed the last Exchange 2007 server they find out that they a LOB application is not working correctly and they need to add a server with Exchange 2007 to allow it working again. The question is: can we add an older Exchange version after we removed the last server from the environment?

A community member on Reddit’s ExchangeServer community recently stated that this would not be possible:

…you won't be able to install a new 2007 server into a forest with 2010 servers once you've removed the last 2007 server…

In a follow-up discussion the poster referred to this article as his source:

Question: Is it possible to install Exchange 2003 or 2007 in a pure Exchange 2010 organization?
Answer: [...] If you have transitioned from Exchange 2007 to Exchange 2010 and the last Exchange 2007 server has already been decommissioned, the answer is again no. You will not be able to install Exchange 2007 at a later time in this organization because it’s now considered a pure Exchange 2010 organization.

This excerpt is from an article in the TechNet Magazine and was written by a well-known Exchange export so I don’t blame the Reddit poster at all. However, with my knowledge of how Exchange works I could not think of any reason why this shouldn’t work and this is the first time I heard of it.

My position on this subject is that there is no reason why you should not be able to add an Exchange 2007 server in a Server 2010 forest after the last Exchange 2007 was removed.

From a technical perspective there’s nothing special happening after you remove the last Exchange 2007 server. The Exchange environment does not keep count of the number of Exchange 2007 servers and there’s no process that implements a blocker when the count reaches <1. Also there’s no such thing as a ‘pure Exchange 2010 organization’, unless you want to say that there are no other Exchange versions than 2010 in the environment.

Time to validate this in a lab:

1. Create a domain controller for a new forest, DFL and FFL set to Windows Server 2003
2. Add a server with Exchange 2007 SP3
3. Add a server with Exchange 2010 SP3
4. Move the mailboxes and OAB generation to Exchange 2010
5. Remove the Exchange 2007 mailbox database and storage group
6. Uninstall Exchange 2007
7. Add a server with Exchange 2007

As expected I was able to add the Exchange 2007 server to the environment without any issues.

Now in a real world environment there can be other factors involved but’s it’s good to understand that your Exchange organization does not add a block when you remove the last server of a certain Exchange version.

Is it possible to install Exchange 2016 on a domain controller?

Can we?

This is a question that comes up every now and then in the technical communities I’m involved in. Is it possible and supported to install Exchange on a domain controller? Often answered incorrectly with a variant of “not supported, unless you run SBS”. The official answer is a bit different: it is supported to install Exchange on a domain controller.

In fact, this is supported since Exchange 2000 and has not changed for every new version up to the recently released Exchange 2016. As we can find in the Exchange 2016 System Requirements page:

Installing Exchange 2016 on directory servers
For security and performance reasons, we recommend that you install Exchange 2016 only on member servers and not on Active Directory directory servers. However, you can't run DCPromo on a computer running Exchange 2016. After Exchange 2016 is installed, changing its role from a member server to a directory server, or vice versa, isn't supported.

So should we?

Supported but not recommended, why is that? There are multiple reasons why you should consider to install Exchange on a member servers.

Firstly both Exchange and an Active Directory domain controller are resource-intensive tasks. For instance, both need to cache data in memory for best performance and will try to claim resources to do so. Running both tasks on the same computer may impact the performance and end-user experience for both AD and Exchange.

Exchange supports an RBAC model which allows an organization to deploy the least amount of privileges to their admins to do their work. However, when an admin needs local administrative access to server which has Exchange installed he gains administrative access to Active Directory too. This would enable the admin to elevate their own privileges which of course is a security risk.

A domain controller with Exchange installed cannot be demoted to remove AD from the computer as long as Exchange is installed, the last version that did support this was Exchange 2000. This may mean that at some later time an organization may find they can’t remove domain controllers from their environment without removing Exchange from the server first. In the recent past this is something that organizations often encountered when they needed to remove Windows Server 2003 domain controllers in order to be able to raise the domain or forest functional level. By the way, you also can’t promote a server with Exchange already installed but that is a much less common scenario.

I briefly touched the subject of security when discussing your admins working on this server, security needs considered with user access in mind too. Some firewall ports need to be forwarded to this server to accept email and give access to OWA/Outlook on the web, ActiveSync, Outlook Anywhere and maybe even legacy protocols as POP3 and IMAP4. Nowadays Exchange is considered secure enough to expose it to the internet directly but do you want to allow hackers to target your domain controller too? Opening firewall ports to allow access from the internet to your domain controller is not something I would call a security best practice.

There’s even more reasons why you should not install on a Exchange server but I won’t do into too much details here. Use your favorite search engine to do a search on this topic and you will find a ton of discussions about this subject.

Yeah, but what if…?

I am aware that for some very small organizations it may not make sense to purchase additional hardware and Windows Server licenses. Certainly not from a cost perspective. With a small load it’s unlikely that performance is going to be an issue and many small business owners will consider the security implications something for large enterprises to be concerned about. After all, Microsoft until recently sold the Small Business Server products that did install Active Directory, Exchange, SharePoint, ISA/TMG and even SQL on a single box. If that was an acceptable solution, why shouldn’t we do that with just AD and Exchange?

Running Exchange on a domain controller is a supported solution and as long as you understand the impact, go ahead and deploy that single server. When performance, security and flexibility play a role and the budget or infrastructure allows for, it’s recommended to deploy a new Windows server and install Exchange on that server.

How to remove the Exchange Autodiscover SCP

Consider a scenario where you moved all your user’s mailboxes to Exchange Online, for instance after a cutover or staged migration and want to remove any dependencies on the local Exchange server. You may find that Outlook still connects to the local Exchange server for Autodiscover lookups, this is because Outlook is hard-coded to query an AD Service Connection Point to locate a server with the Autodiscover service. When this fails Outlook falls back to the next DNS based methods or uses a local XML file.

Exchange Management Shell

There are multiple ways to prevent Outlook from contacting the local Exchange server first, some of them make more sense than others. The preferred way is to use the Exchange Management Shell to clear the entry for the Client Access server from the SCP.

Set-ClientAccessServer –Identity ServerName -AutoDiscoverServiceInternalUri $null

This removes the SCP entry for this Exchange server.

ADSIEdit

If the above method can no longer be used a low level AD editor as EDSIEdit can be used to remove the SCP manually. The full path of the SCP is:

CN=ServerName,CN=Autodiscover,CN=Protocols,CN=ServerName,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=OrganizationName,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=DomainName,DC=Suffix

This object to remove has the Class type serviceConnectionPoint.

Alternative methods

One of the above steps is basically all you need to do. Alternatives include adding an ExcludeScpLookup value to the \Autodiscover registry key (KB article) and some articles even let you remove the Autodiscover virtual directory from IIS. This will of course make Outlook unable to query the local Exchange server for Autodiscover but why should you if you can simply remove the SCP.

How to verify?

As always, the proof is in the pudding. Use the Outlook test E-mail AutoConfiguration feature to verify the clients behavior. We’re specifically interested in the Log tab where we should see that Outlook is no longer able to query the SCP to obtain the Autodiscover url.

To start the Test E-mail AutoConfiguration tool, follow these steps:

1. Start Outlook.
2. Hold down the Ctrl key, right-click the Outlook icon in the notification area, and then click Test E-mail AutoConfiguration.
3. Verify that the correct email address is in the E-mail Address box.
4. In the Test E-mail AutoConfiguration window, click to clear the Use Guessmart check box and the Secure Guessmart Authentication check box.
5. Click to select the Use AutoDiscover check box, and then click Test.

Earlier I wrote a short article about this tool, unfortunately available in Dutch only: Autodiscover testen met Outlook. But you’ll get the general idea. Focus on the first or third tab when you’re interested in the Autodiscover results, read the Log tab when you're interested in the Autodiscover process.

MySQL Workbench and ClearDB: "SSL Connection error: unable to get private key file"

When trying to access your ClearDB hosted databases with MySQL Workbench you may run into this error message:

SSL Connection error: Unable to get private key

In the MySQL Workbench connection settings on the SSL tab we need to select three certificate files, the CA certificate, the client certificate and the client key file.

These files can be obtained from the ClearDB database dashboard:

The issue is being caused by the client private key file, the one with 'key' in the name. In order to allow Workbench to connect to the database the password needs to be stripped from this file first. One way to do this is to use OpenSSL:

openssl rsa -in c092bb5f82771e-key.pem -out c092bb5f82771e-key.pem-nopassw.pem

Now use the new file in the Workbench connection settings and try again, Workbench should now be able to connect to your database.

Source

How to view the last monitor response in NetScaler 10.5

When a monitor fails it can be very helpful to check the last result it received. The NetScaler 10.5 web interface does present this information, but you have to know where to look:

Open de properties of the Service Group, expand the Service Group Members, select a Service and click Monitor Details.

The result can be found under the Last Response column:

How to create a Citrix NetScaler LB vserver without an IP address

When configuring a Citrix NetScaler to load balance Exchange 2013 or 2016 you may want to create a load balancing virtual server without an IP and port combo. The reason is that you're going to bind the load balancing virtual server will to a content switching virtual server, clients won't access the lb vserver directly.

Many blog posts as well as Citrix documents as for instance their most recent Exchange whitepaper Microsoft Exchange 2013 with NetScaler: Authentication and Optimization tell you to select Not Directly Addressable. Unfortunately the Directly Addressable check box is no longer available in NetScaler 10.5 and newer.

Well, the feature is still available but has been renamed and moved to another location. If you're using the web interface, choose Non Addressable from the IP Address Type drop-down list no create a virtual server which is not directly accessible.

Maybe it's just me, but I could not found the cmdline alternative in the lb vserver section of the NetScaler Command Reference. So I create one with the web interface and looked it up in the config. Apparently all we need to do is enter the IP address and port number as 0.0.0.0 0.

add lb vserver my_lb_vserver ssl 0.0.0.0 0

In summary, the checkbox for Directly Addressable is replaced with the Non Addressable option in the IP Address Type drop-down list. If you're using the cmdline, set the IP address and port number as 0.0.0.0 0.

What does the new 'Phishing scam' button in Outlook on the web do?

The website formerly known as OWA

In case you misses it, Microsoft recently rebranded Outlook Web App (OWA) to Outlook on the web. I'm sure the marketing people had very good reasons to do so, but the OWA acronym has become a household name since it was introduced in Exchange 5.0. So technically speaking we should use the term Outlook on the web when discussing OWA on Exchange 2016 or in Exchange Online however I will use OWA a lot for probably a very long time. Enough about branding, let's discuss the actual topic.

The new Outlook on the web

Over the past couple of months Microsoft rolled out an updated version of OWA/Outlook on the web to all Office 365 tenants. The new features include Pin, Sweep and the Undo button:

One feature that was added recently, in fact it's even absent from the above screenshot Microsoft used in their announcement, is the new Phishing scam action. This can be found in the actions drop-down list next to Reply all in the reading pane and under Junk in the top actions bar:

The question of course is what exactly happens when you apply this action to your message. At the time of writing there was no public documentation for Exchange or Exchange Online that describes the behavior of this action. However, the button was implemented in the consumer service Outlook.com earlier and is mentioned in the article Email and web scams: How to help protect yourself.

So the Phishing scam action deletes the item and marks the sender as unsafe, just as the Junk action, but it reports the message to Microsoft too. They then use the reported phishing scam emails to improve their filtering techniques to prevent them from arriving in your user's Inbox.

Should I use it?

Definitely encourage your users to use the Phishing scam action on phishing emails, this helps Microsoft fight these scams and make your user's Inbox a safer place. Technology can certainly help here, but in the end user education is the most important and effective way to reduce the risks of being scammed.

Just Office 365?

Interestingly enough the new feature is not in the on-premises Exchange 2016 which was just released this week. From an architectural perspective I see no reason why Microsoft shouldn’t implement the feature in the on-premises product too. After all this feature is nothing more than a slightly adapted version of the Junk feature that is already on-premises Exchange.

My expectation is that we see the Phishing scam action appear in one of the next Cumulative Updates for Exchange 2016. The same applies to Outlook 2006, I expect the same feature to be added in an upcoming update too.

Office 2016 update branches and how to force an upgrade for Office 365 ProPlus

Last week Microsoft released Office 2016, the most recent edition of their productivity suite including Word, Excel and Outlook. What does this mean for customers who consume Office as part of the Office 365 ProPlus subscription? Why was my Office version upgraded without any warning? Or why am I still on Office 2013?

Update branches

Microsoft introduced ‘update branches’ to release updates to three types of customers:

• Current Branch: Monthly updates with the latest features
• Current Branch for Businesses: Quarterly updates
• Current Branch for Businesses First Release: Test the next current branch before it is released

Read more about release branches in the TechNet Technical Library: Overview of update branches for Office 365 ProPlus or this excellent write-up by Perficient’s Joe Palarchio How Is The Office 2016 Release Relevant To My Office 365 Users?

I want it now!

So if you have Office 365 ProPlus and your Office 365 plan is not Home or Business Premium but for instance Enterprise, you may want to know if you can force an upgrade to Office 2016 on your computer. To achieve that we need to switch out Office install from Current Branch for Businesses to Current Branch. In an enterprise the admins would configure this with a GPO template for Office 2016 or with the Office Deployment Tool when Office is installed on the user’s computer.

The latter is very easy to upgrade Office 365 ProPlus on a single computer to, just follow this procedure.

1. Download the Office Deployment Tool and extract the files to a temporary location, for instance C:\Office.
   
2. Make a backup of the configuration.xml file and edit the contents to something similar to this:
   
   <Configuration>
       <Add SourcePath="c:\Office\" OfficeClientEdition="64" Branch="Current">
           <Product ID="O365ProPlusRetail">
           <Language ID="en-us" />
           </Product>
       </Add>
       <Updates Enabled="TRUE" Branch="Current" />
       <Display Level="Full" AcceptEULA="TRUE" />
   </Configuration>
   
   Note that the value for Branch is set to Current. Other valid values are Business or Validation (First Release).
3. Execute .\setup.exe /download c:\Office\configuration.xml to download the Office files to your local computer, this may take a while.
4. Next start the installation with .\setup.exe /configure c:\Office\configuration.xml

During the installation of Office you will be prompted to save your work and close any opened Office programs.

Enjoy Office 2016!

Using Exchange 2007? Do not upgrade to Outlook 2016

In case you missed it, this week Microsoft released the new Office suite: Office 2016. Of course the latest version includes an updated version of Outlook too. Many people are eager to explore the new version of Office and others found their Office was automatically updated to Office 2016. From an Exchange perspective it’s important to be aware of the fact that Outlook 2016 does not support connections to Exchange 2007.

Customers who upgrade to Outlook 2016 and try to connect to their Exchange 2007 mailbox see this error message presented: The resource that you are trying to use is located on an unsupported version of Microsoft Exchange. Contact your e-mail administrator for assistance

This information can of course be found in the Office System Requirements, but you have to look for it:

Technically speaking there are workarounds available such as have Outlook 2016 access the Exchange 2007 server with the IMAP4 or POP3 protocol, but this of course introduces serious limitations and is not comparable with connecting through a MAPI or Outlook Anywhere connection.

Microsoft released an article which describes this issue and instructs Office 365 Personal, Office 365 Home, or Office 365 University users who's Outlook was automatically upgraded to Outlook 2016 to contact Microsoft Support to assist with a roll-back to Office 2013. See Error: Unsupported version of Microsoft Exchange error when you use Outlook 2016 to connect an Exchange 2007 account.

By the way, if your organization is still using Exchange 2007 you may want to look into an upgrade to a more recent Exchange version or Exchange Online. Exchange 2007 is in the Extended Support phase since 2012 and will go out of support in April 2017.

Upgrade to Windows 10: Hyper-V no longer installed

Update: It seems the upgrade process has overwritten the default locations where my virtual machines were stored. If you did not yet upgrade and care for your Hyper-V VMs, be very careful. Make sure you move the virtual machines to a safe location first.

After upgrading my Surface 3 Pro to Windows 10 I noticed that Hyper-V was no longer installed. I spend some time in the new Settings app but could not find a way to enable Hyper-V again.

Fortunately this issue is very easy to fix. Open Control Panel and navigate to Programs, Turn Windows features on or off. Select Hyper-V and click OK to continue.

But before you do this, please check the full list and select or deselect any other features you want to install or remove.

How to access the Exchange 2016 ECP/EAC with a mailbox on 2013 or 2010?

So you added the first Exchange 2016 Preview server to your lab and now you want to access the Exchange admin center to configure your server. When you try to access https://<Exchange2016MailboxServer>/ecp and you enter your credentials you may see a ‘500 Unexpected Error’ or end up with the 2010 or 2013 version of the ECP. This is because Exchange 2016 by default tries to present the version of ECP that corresponds with the version of Exchange where your mailbox is hosted on.

To access the Exchange 2016 admin center while your mailbox is on an older version, append the string ?ExchClientVer=15.1 to your url. For instance https://<Exchange2016MailboxServer>/ecp?ExchClientVer=15.1.

Sounds familiar? That’s because the same procedure applied to Exchange 2013. Please note that the major version number of Exchange 2016 is 15.1, not 16 as you may have guessed.

Exchange 2016 Preview released!

Earlier today I wrote about some Exchange 2016 content that appeared on TechNet and now it’s obvious why: Microsoft released the Preview version of Exchange 2016. For those of you who attended Ignite this year the announcement will bring not much new. The architecture has been simplified (CAS and Mailbox roles integrated), OWA supports in-line editing and viewing of Office attachments, search has improved (again) and the Hybrid Configuration Wizard now runs from Office 365.

A feature that was not shared earlier is the auto-expanding archive mailbox, after the first 100 GB Exchange will automatically add archive mailboxes in 50 GB increments. I guess this will be interesting for some on-premises users but this is obviously a feature targeted at Exchange Online.

A download link and more information can be found in the article at the Exchange Team Blog.

PowerShell one-liner: How to query certificates in the certificate store?

PowerShell uses providers and drives to provide a consistent way to work with items in the file system, Active Directory, the registry, in applications and even the certificate store.

Recently I started using PowerShell to find the thumbprint of an installed certificate, for instance when I need that value to enable a certificate for Exchange services. To do this we can use the Cert: drive, navigate to the Local Computer store and then query the items in My, this is the Personal container you’re probably familiar with from working with the Certificates MMC snap-in.

dir cert:\localmachine\my

Where dir is of course an alias for Get-ChildItem.

Microsoft publishes Exchange 2016 documentation in Technical Library

With the Exchange 2016 Preview scheduled for the summer of 2015 (now!) the first documentation has been published to the Exchange Technical Library on TechNet. Of course the available content is still limited and new content will be added towards RTM.

Check it out here: Exchange Server 2016

Exchange 2013 CU install fails because the certificate is expired

This issue was recently brought up in a community and today I ran into the same issue myself. An Exchange 2013 CU installation is in progress and after Setup removed the existing installation files, it fails while installing the Transport service of the Mailbox role:

Error:
The following error was generated when "$error.Clear();
          Install-ExchangeCertificate -services IIS -DomainController $RoleDomainController
          if ($RoleIsDatacenter -ne $true -And $RoleIsPartnerHosted -ne $true)
          {
            Install-AuthCertificate -DomainController $RoleDomainController
          }
        " was run: "System.Security.Cryptography.CryptographicException: The certificate is expired……

Want happens here is quite easy to understand. As part of the CU installation Setup tries to enable the SSL certificate to the IIS service. This fails because the Valid To date on the certificate has passed, the certificate is no longer valid.

Easy, we simply replace the cert right? Well, remember that Exchange already removed the existing install? We have no access to the EMS at this point so we need Setup to finish the install before we can replace the certificate the proper way.

A silly but effective workaround to achieve this goal is to change the system time of the server to a date that falls in the range where the certificate was still valid.

Note: Make sure you (temporarily) disable the time synchronization feature of your hypervisor and the Windows Time service, or else it will change the time back in no time. :)

Now you can restart the CU installation, it will automatically detect the failed attempt and offers to continue the process.

When the CU installation has finished, enable the Windows Time service and/or the time sync feature of your hypervisor and observe the clock moving back to the correct time. Now would be a great time to fire up EMS and replace the SSL certificate with a new one. Reboot the box as best practice after installing a CU anyway and check the health of the server to verify if everything is working as it should.

So if your reading this you probably started your lab servers after a long time, just like I did. If you ran into this issue in a production environment, it's important to investigate why you ran with an expired certificate anyways. And if your certificate has expired, this article shows why you should replace it before you perform any maintenance on the server.

Microsoft updates the Office 365 portal

Microsoft has done a lot of work in their Office 365 Portal over the past few years, anyone who remembers the BPOS experience will agree. The experience we see today is very consistent and you need to keep an eye on the address bar of your browser to notice that you actually switched to a different website. It’s all modern, clean and very white-blue.

An area for improvement is the end-user self-service portal, the section that can be accessed by clicking on the gear icon in the top right corner of the screen. Recently Microsoft started updating this section too. First thing the user will notice is the gradient bar on top of the page and the centered items, in the previous version of the portal the items were aligned to the left of the page.

When a user clicks on a section head the area expands and allows the user to make changes without leaving the page. For instance when the user clicks on Language, the option to select the language appears:

There are two exemptions to this principle, that's the Password and Software sections. I expect those sections to be revised too somewhere in the near future. Many admins are waiting for the option to remove the Password section from this portal, let's hope we see this added soon.

Is your 'free' Exchange hybrid key really free?

And I'm not talking about Willy or Nelson Mandela, I mean free as in at 'at no additional cost'. There are numerous sources on the internet stating customers can obtain a free key for their hybrid server. What most articles forget to mention is that the license restrictions make this license free for just a very small subset of all customers.
And it's not just blog posts of independent writers, I heard Microsoft employees state the same while visiting customers and in presentations on tech conferences as MEC and TechEd. And even their new Exchange Hybrid Product Key Distribution wizard (http://aka.ms/hybridkey) does not mention all requirements.

In fact there are three major requirements to obtain the license key for free:

• You have an existing, non-trial, Office 365 Enterprise subscription
• You will not host any on-premises mailboxes on the Exchange 2013 or Exchange 2010 SP3 server on which you apply the Hybrid Edition product key.

And the one I want to emphasize:

• You currently do not have a licensed Exchange 2013 or Exchange 2010 SP3 server in your on-premises organization

So if you're running licensed servers with Exchange 2010 in your environment, that Exchange 2013 server you want to deploy for hybrid is not free! In other words, the hybrid server license key is free if you're running Exchange 2007 or Exchange 2003 and have licenses for just those versions of Exchange.
In all other situations you will need to license your hybrid servers properly.
These license limitations can be found in the following KB article: How to obtain an Exchange Hybrid Edition product key for your on-premises Exchange 2007 or Exchange 2003 organization.

Update: Office 365 migrations and the 'Delete‎()‎ is not supported on a read-only session' error

If you're performing a Staged or Cutover migration to Exchange Online you may run into the following error:

Error: MigrationPermanentException: An error occurred while running Get-MergeRequest -Identity : Delete‎()‎ is not supported on a read-only session. --> Delete‎()‎ is not supported on a read-only session.

Many customers reported this issue in the last couple of days. Moderators in the Office 365 Support Forums confirmed that Microsoft is aware of the issue in their backend and is still investigating. Unfortunately the Service Health dashboard does not make mention of this issue.

If you run into this issue you could try to stop and delete the migration batch, delete the created Office 365 users and restart the process. Some people reported their migration to succeed now. Others are still seeing the same issue in their migration batch.

To make sure that Microsoft has a good understanding of the scale of the issue, please open a Service Request if you're impacted too. And keep an eye on the discussion thread in the support forums to see if there's any progress made in resolving this issue.

Update July 14th 2015

Microsoft confirmed they implemented a fix in their environment, but it may take some time before it applies to all tenants. To find your tenant version, connect to Exchange Online with PowerShell first. Then query the version number:

Get-OrganizationConfig | ft AdminDisplayVersion

The fix has been implemented in version 15.1.234, so if your tenant is on that version or newer you can restart the migration batch and expect it to no longer fail. If your tenant is still on an older version I'm afraid you just have to wait a little more.

How to remove the Change Password link from the Office 365 portal

Microsoft did an amazing job with their online services. I remember the web based management in the BPOS days, not a pleasant experience for both end users and admins. The current Office 365 portal is much improved and receives new updates every couple of months. Unfortunately there's one thing bothering many Office 365 admins and confusing even more end users.

In the Office 365 settings page (url: https://portal.office.com/EditProfile15.aspx), where the users can change their personal settings, is a Password link. This works great for cloud users, user accounts that have not been synced from an on-premises Active Directory, but not so well for synced users with either Password Synchronization or Identity Federation (ADFS).

When those type of users click this link to update their password the will receive an error:

Sorry, you can't change your password here. Follow the steps recommended by your organization or ask your admin for help.

So can we could remove the link, at least for our synced users? The answer is no, you can't currently do that. Recently I made a request through the Office 365 Support channel and so did other customers, leading to a Design Change Request (DRC). Unfortunately this DCR was declined by the Product Group because this change is not feasible with the current version of the portal.

If you want to be sure that the need for this change is under Microsoft's attention, please submit your feedback with the form on this page.

Azure AD Sync doesn't warn if scheduled task is enabled on Server 2008/2008 R2

After running the Azure Active Directory Sync Services (AADSync) configuration wizard, a scheduled task is created to run a sync job every three hours. When an admin starts the wizard again to make changed to the configuration a warning is thrown to disable the scheduled task and forced to restart the wizard. This is to prevent configuration changes to be made while an actual sync could be in progress. This check does not work with Server 2008 and Server 2008 R2.

Under the hood AADSync uses the Get-ScheduledTask cmdlet to determine the status of the scheduled task. Unfortunately this cmdlet was introduced in Server 2012 and Windows 8, it's not available on Server 2008 and Server 2008 R2. Both older versions of Windows Server are on the list of supported operating systems.

So what happens if you have AADSync installed on Server 2008 or 2008 R2 and start the wizard again? It does not warn you to disable the scheduled task first and allows you to change the configuration while a sync could be in progress. While the chances of that happening are relatively small with the three hour interval, this obviously is not something we want to happen. The application log shows event id 906 from source AzureActiveDirectoryDirectorySyncTool:

IsSchedulerEnabled() failed, assuming FALSE: Details: System.Management.Automation.CommandNotFoundException: The term 'Get-ScheduledTask' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.

The error message is self-explanatory, the IsScheduleEnabled function tries to use the Get-ScheduledTask cmdlet which is not available on this server. And the function assumes that the task is disabled, this is why we're no longer prevented from making configuration changes while the task is enabled. This behavior was noted with AADSync version 1.0.494.0501, the most recent version at this moment.

What does this mean for you if you're running AADSync on an older operating system? You should remember to verify that you disable the scheduled task before starting the configuration wizard, keep in mind that the wizard will not be able to check this and warn you if the task is still enabled.

Although Server 2008 and 2008 R2 are supported operating systems for AADSync I suspect Microsoft did not actually test the software on those operating systems. I brought the issue under their attention through Office 365 Support, an experience I wouldn't wish to my worst enemy. To be continued…

Short rant about Exchange KB3035227 and MobileSyncRedirectBypass

I need to vent, that's all. I'm working with a customer to prepare for the installation of Exchange 2010 SP3 Update Rollup 9. The list of fixed issues is very short and counts only six bullets and a remark about DST updates:

The amazing Rhoderick Milne wrote a post about Exchange 2013 CU and mentioned an ActiveSync issue which involved UR 9 for Exchange 2010 SP3 too.

Please see an emerging issue with ActiveSync after deploying Exchange 2010 SP3 RU9 or Exchange 2013 RU8.

The article he refers to is KB3035227: Some Android devices can't set up an Exchange account after you install Exchange Server 2010 SP3 RU9 or Exchange Server 2013 CU8 So there is a known issue with UR9 but it is not included in the Known Issues section of Update Rollup 9 for Exchange Server 2010 Service Pack 3. That's annoying but well, I found the article so let's read...

So we're talking about adding a new ActiveSync (yes, that's one word and not Active Sync like the article says) account and Android fails to complete the Autodiscover process.

In the Cause section is a little bit of explanation about this issue:

Exchange Server 2010 SP3 RU9 and Exchange Server 2013 CU8 contain a new MobileSyncRedirectBypass feature in the Autodiscover service for Android devices. By default, this feature is enabled. Some versions of Android may not understand this feature.

Maybe I'm a bad reader but I still don't understand. What is this "new MobileSyncRedirectBypass feature in the Autodiscover service"? And where in the Autodiscover process will it send a 451 redirect message to the client? And what conditions would trigger this changed behavior?

Luckily there's a link for more information: Exchange ActiveSync Autodiscover is incomplete. The link goes to an article about the Android Activesync implementation and explains that Android does not support the HTTP redirect method which involves a 302 redirect. On the same page we can read that Android does support the 451 redirect. So the "more information" link doesn't answer questions, it raises new questions.

Now have a look at the instructions to disable this new feature. It starts with:

If the MobileSyncRedirectBypass feature is causing the problem, you can turn it off...

Wait, how can we determine that the MobileSyncRedirectBypass feature is causing the problem? Anyway, let's look at the steps:

1. Locate the web.config file for the Autodiscover protocol:

   1. For Exchange Server 2013 MBX, the file is in the following location:

      %ExchangeInstallPath%\ClientAccess\Autodiscover

   2. For Exchange Server 2010 CAS, the file is in the following location:

      %ExchangeInstallPath%\ClientAccess\Autodiscover

2. Open the web.config in Notepad, and then change the existing string from "true" to "false."

What existing string? My guess would be MobileSyncRedirectBypass but there's no such string in the file:

So this article fails to explain the issue or what's causing the issue and the resolution steps are incomplete. And the worst part?

Apparently this is the 4th major revision of the article!

End of rant. Does it make me feel better? Not really. I promise to write a more constructive post once I obtained more information about this issue.

Does Exchange 2013 CU9 include schema updates?

The recent announcement of the latest Cumulative Update for Exchange 2013 said this about schema changes:

Cumulative Update 9 may include Exchange related updates to the Active Directory schema and Exchange configuration when compared with the version of Exchange 2013 you have currently deployed.

So does CU9 include a schema update or doesn't it? The answer is hidden in the "the version of Exchange 2013 you have currently deployed" part.

First let's take a look at how we can find out what version our schema is. To find the Exchange schema version we need to query the rangeUpper property of CN=ms-Exch-Schema-Version-Pt,cn=schema,cn=configuration,DC=domain,DC=local (replace DC=domain,DC=local with the name of your root domain).

In PowerShell:

$RootDSE= ([ADSI]"").distinguishedName
([ADSI]"LDAP://CN=ms-Exch-Schema-Version-Pt,CN=Schema,CN=Configuration,$RootDSE").rangeUpper

In an Exchange forest with CU9 installed the value returned by this query is 15312. There are several online resources which keep track of all schema versions, for instance this one by Michel de Rooij or this one by Todd Nelson.

So apparently CU7 was the latest update to include schema updates. This means the CU9 will perform a schema update when upgrading from CU6 or older and of course when CU9 is the first Exchange 2013 version you deployed in your environment. If you're upgrading from CU7 or CU9 the schema will not be updated.

Exchange 2010 in Extended Support phase, last scheduled UR released today

Exchange 2010 entered the Extended Support phase earlier this year. Basically this means that only security updates will be released to all customers. New and improved features will no longer be added and non-security bugs will be not always be fixed.

Exchange 2010 is a little bit special because it's currently the most widely deployed version of Exchange. And why not, the HA model with the DAG works like a charm, the product is stable and mature and has all the features you can think of. Of course the web interface is looks a bit outdated but it works very well.

The question is if and when customers should upgrade from something that runs pretty well. There is some similarity with Exchange 2003 in the Exchange 2007 and early 2010 timeframe, the software was outdated but it worked very well. This is why so many customers kept running Exchange 2003 until and past the end of Extended Support phase.

Today Microsoft announced Update Rollup 10 for Exchange 2010 SP3. The list with resolved issues counts only 7 bullets.

Very interesting is this little remark at the bottom of the announcement on the Exchange team blog:

Update Rollup 10 is the last scheduled release for Exchange Server 2010. Both Exchange Server 2010 and Exchange Server 2007 are in extended support and will receive security and time zone fixes on-demand on a go-forward basis.

So if you're running 2010, now would be a great time to start planning your upgrade to Exchange 2013, which is rock solid with the 9th CU released today, or Exchange 2016 which is scheduled for release later this year.