How to remove the Change Password link from the Office 365 portal

Microsoft did an amazing job with their online services. I remember the web based management in the BPOS days, not a pleasant experience for both end users and admins. The current Office 365 portal is much improved and receives new updates every couple of months. Unfortunately there's one thing bothering many Office 365 admins and confusing even more end users.

In the Office 365 settings page (url: https://portal.office.com/EditProfile15.aspx), where the users can change their personal settings, is a Password link. This works great for cloud users, user accounts that have not been synced from an on-premises Active Directory, but not so well for synced users with either Password Synchronization or Identity Federation (ADFS).

When those type of users click this link to update their password the will receive an error:

Sorry, you can't change your password here. Follow the steps recommended by your organization or ask your admin for help.

So can we could remove the link, at least for our synced users? The answer is no, you can't currently do that. Recently I made a request through the Office 365 Support channel and so did other customers, leading to a Design Change Request (DRC). Unfortunately this DCR was declined by the Product Group because this change is not feasible with the current version of the portal.

If you want to be sure that the need for this change is under Microsoft's attention, please submit your feedback with the form on this page.

Azure AD Sync doesn't warn if scheduled task is enabled on Server 2008/2008 R2

After running the Azure Active Directory Sync Services (AADSync) configuration wizard, a scheduled task is created to run a sync job every three hours. When an admin starts the wizard again to make changed to the configuration a warning is thrown to disable the scheduled task and forced to restart the wizard. This is to prevent configuration changes to be made while an actual sync could be in progress. This check does not work with Server 2008 and Server 2008 R2.

Under the hood AADSync uses the Get-ScheduledTask cmdlet to determine the status of the scheduled task. Unfortunately this cmdlet was introduced in Server 2012 and Windows 8, it's not available on Server 2008 and Server 2008 R2. Both older versions of Windows Server are on the list of supported operating systems.

So what happens if you have AADSync installed on Server 2008 or 2008 R2 and start the wizard again? It does not warn you to disable the scheduled task first and allows you to change the configuration while a sync could be in progress. While the chances of that happening are relatively small with the three hour interval, this obviously is not something we want to happen. The application log shows event id 906 from source AzureActiveDirectoryDirectorySyncTool:

IsSchedulerEnabled() failed, assuming FALSE: Details: System.Management.Automation.CommandNotFoundException: The term 'Get-ScheduledTask' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.

The error message is self-explanatory, the IsScheduleEnabled function tries to use the Get-ScheduledTask cmdlet which is not available on this server. And the function assumes that the task is disabled, this is why we're no longer prevented from making configuration changes while the task is enabled. This behavior was noted with AADSync version 1.0.494.0501, the most recent version at this moment.

What does this mean for you if you're running AADSync on an older operating system? You should remember to verify that you disable the scheduled task before starting the configuration wizard, keep in mind that the wizard will not be able to check this and warn you if the task is still enabled.

Although Server 2008 and 2008 R2 are supported operating systems for AADSync I suspect Microsoft did not actually test the software on those operating systems. I brought the issue under their attention through Office 365 Support, an experience I wouldn't wish to my worst enemy. To be continued…

Short rant about Exchange KB3035227 and MobileSyncRedirectBypass

I need to vent, that's all. I'm working with a customer to prepare for the installation of Exchange 2010 SP3 Update Rollup 9. The list of fixed issues is very short and counts only six bullets and a remark about DST updates:

The amazing Rhoderick Milne wrote a post about Exchange 2013 CU and mentioned an ActiveSync issue which involved UR 9 for Exchange 2010 SP3 too.

Please see an emerging issue with ActiveSync after deploying Exchange 2010 SP3 RU9 or Exchange 2013 RU8.

The article he refers to is KB3035227: Some Android devices can't set up an Exchange account after you install Exchange Server 2010 SP3 RU9 or Exchange Server 2013 CU8 So there is a known issue with UR9 but it is not included in the Known Issues section of Update Rollup 9 for Exchange Server 2010 Service Pack 3. That's annoying but well, I found the article so let's read...

So we're talking about adding a new ActiveSync (yes, that's one word and not Active Sync like the article says) account and Android fails to complete the Autodiscover process.

In the Cause section is a little bit of explanation about this issue:

Exchange Server 2010 SP3 RU9 and Exchange Server 2013 CU8 contain a new MobileSyncRedirectBypass feature in the Autodiscover service for Android devices. By default, this feature is enabled. Some versions of Android may not understand this feature.

Maybe I'm a bad reader but I still don't understand. What is this "new MobileSyncRedirectBypass feature in the Autodiscover service"? And where in the Autodiscover process will it send a 451 redirect message to the client? And what conditions would trigger this changed behavior?

Luckily there's a link for more information: Exchange ActiveSync Autodiscover is incomplete. The link goes to an article about the Android Activesync implementation and explains that Android does not support the HTTP redirect method which involves a 302 redirect. On the same page we can read that Android does support the 451 redirect. So the "more information" link doesn't answer questions, it raises new questions.

Now have a look at the instructions to disable this new feature. It starts with:

If the MobileSyncRedirectBypass feature is causing the problem, you can turn it off...

Wait, how can we determine that the MobileSyncRedirectBypass feature is causing the problem? Anyway, let's look at the steps:

1. Locate the web.config file for the Autodiscover protocol:

   1. For Exchange Server 2013 MBX, the file is in the following location:

      %ExchangeInstallPath%\ClientAccess\Autodiscover

   2. For Exchange Server 2010 CAS, the file is in the following location:

      %ExchangeInstallPath%\ClientAccess\Autodiscover

2. Open the web.config in Notepad, and then change the existing string from "true" to "false."

What existing string? My guess would be MobileSyncRedirectBypass but there's no such string in the file:

So this article fails to explain the issue or what's causing the issue and the resolution steps are incomplete. And the worst part?

Apparently this is the 4th major revision of the article!

End of rant. Does it make me feel better? Not really. I promise to write a more constructive post once I obtained more information about this issue.

Does Exchange 2013 CU9 include schema updates?

The recent announcement of the latest Cumulative Update for Exchange 2013 said this about schema changes:

Cumulative Update 9 may include Exchange related updates to the Active Directory schema and Exchange configuration when compared with the version of Exchange 2013 you have currently deployed.

So does CU9 include a schema update or doesn't it? The answer is hidden in the "the version of Exchange 2013 you have currently deployed" part.

First let's take a look at how we can find out what version our schema is. To find the Exchange schema version we need to query the rangeUpper property of CN=ms-Exch-Schema-Version-Pt,cn=schema,cn=configuration,DC=domain,DC=local (replace DC=domain,DC=local with the name of your root domain).

In PowerShell:

$RootDSE= ([ADSI]"").distinguishedName
([ADSI]"LDAP://CN=ms-Exch-Schema-Version-Pt,CN=Schema,CN=Configuration,$RootDSE").rangeUpper

In an Exchange forest with CU9 installed the value returned by this query is 15312. There are several online resources which keep track of all schema versions, for instance this one by Michel de Rooij or this one by Todd Nelson.

So apparently CU7 was the latest update to include schema updates. This means the CU9 will perform a schema update when upgrading from CU6 or older and of course when CU9 is the first Exchange 2013 version you deployed in your environment. If you're upgrading from CU7 or CU9 the schema will not be updated.

Exchange 2010 in Extended Support phase, last scheduled UR released today

Exchange 2010 entered the Extended Support phase earlier this year. Basically this means that only security updates will be released to all customers. New and improved features will no longer be added and non-security bugs will be not always be fixed.

Exchange 2010 is a little bit special because it's currently the most widely deployed version of Exchange. And why not, the HA model with the DAG works like a charm, the product is stable and mature and has all the features you can think of. Of course the web interface is looks a bit outdated but it works very well.

The question is if and when customers should upgrade from something that runs pretty well. There is some similarity with Exchange 2003 in the Exchange 2007 and early 2010 timeframe, the software was outdated but it worked very well. This is why so many customers kept running Exchange 2003 until and past the end of Extended Support phase.

Today Microsoft announced Update Rollup 10 for Exchange 2010 SP3. The list with resolved issues counts only 7 bullets.

Very interesting is this little remark at the bottom of the announcement on the Exchange team blog:

Update Rollup 10 is the last scheduled release for Exchange Server 2010. Both Exchange Server 2010 and Exchange Server 2007 are in extended support and will receive security and time zone fixes on-demand on a go-forward basis.

So if you're running 2010, now would be a great time to start planning your upgrade to Exchange 2013, which is rock solid with the 9th CU released today, or Exchange 2016 which is scheduled for release later this year.

Exchange Best practice: Increase the Application log file

Exchange is known as a chatty application when it comes to the Application log. And even while most of the information nowadays is written to the crimson channel, the classic Application still is a great source of information when you need to troubleshoot an issue.

The default settings of the Application log are a maximum size of 16 MB and older events are then overwritten when the log has reached that size. Now this may seem a lot, but can be be only a few hours of information for a busy server in a large environment. Good luck troubleshooting that issue that happened yesterday.

I know some of you are already doing this by default on a new Exchange server deployment, but you should definitely increase the log file size to a larger value. This can be done with the Limit-Eventlog cmdlet.

Limit-EventLog -LogName Application -MaximumSize 100MB

Or on all your Exchange servers at once:

Get-ExchangeServer | % { Limit-EventLog -LogName Application -MaximumSize 100MB -ComputerName $_.name  }

Well, you get the idea. More information: Limit-Eventlog

Exchange 2013 and the MS15-064 vulnerability

Microsoft released a security update for Exchange 2013 to fix a new vulnerability. MS15-064 has a severity rating of 'Important'.

Download the update for Exchange 2013 CU8 and Exchange 2013 SP1. More information: Vulnerabilities in Microsoft Exchange Server Could Allow Elevation of Privilege (3062157)

How to disable IPv6 on a Windows computer

Let me start with this:

Best practice is to leave IPv6 enabled on all computers.

However, if there's a valid reason to disable IPv6, it's essential to do this the proper way. An example of such a reason is coexistence of Exchange 2007 and 2013 were the official documentation instructs the admin to disable IPv6 on the Exchange 2007 server.

Simply disabling the IPv6 protocol on your network adapter settings may seem the easiest, but is known to break stuff. Instead use the DisabledComponents registry key to control the IPv6 status. For instance, to disable the entire IPv6 stack on a Windows computer you can set the value to ff:

Set-ItemProperty HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip6\Parameters DisabledComponents 0xff -type dword

For information, read How to disable IPv6 or its components in Windows

"Logjam" and KEMP LoadMaster: not vulnerable

KEMP Technologies releases a statement about their KEMP LoadMaster line of application delivery controllers and the recent Logjam vulnerability, also known as CVE-2015-4000.

In short, the LoadMaster products do not use the weak DHE_EXPORT cipher suites and are NOT vulnerable for Logjam exploits. Reed the full statement here.

New EOP connector validation fails when connector is disabled

Recently Microsoft announced a new feature for creating connectors in Exchange Online Protection: Announcing a new way to create connectors in Office 365. What it basically does is adding a few steps to the wizard after an admin creates an outbound connector. In these steps the configuration of the connector is being validated by sending a test email.

Validation is mandatory and a connector can only be saved after you performed validation, whether validation failed or succeeded. I think this is a great feature because configuration mistakes can be easily made and often result in huge queues filling up or lot of NDRs being sent.

Now unfortunately there is an issue with the current implementation. Consider a scenario where you need to prepare a connector but you're not ready to enable the connector and actually modify the mail flow configuration. When you for instance need approval first or wait for a maintenance window to activate the changed configuration.

The wizard anticipates this scenario and allows you to uncheck the "Turn it on" option on the second page of the wizard:

Before the latest update this would result in a disabled but configured connector. Currently this results in a failed validation, even if the settings of the connector are all correct.

In this example I created an outbound connector from Office 365 to a Partner Organization, used the default options where TLS is applied and added the Hotmail.com domain to the connector. After validation the Send test mail step fails:

The error message is "The domain of the recipient is not configured as part of connector". The Detailed Log doesn't provide any details about why the test failed. In fact it didn't even fail, the test email was received successfully:

This issue is being caused by the fact we choose not to enable this connector. If we would not have removed the checkmark and allow EOP to create and enable the connector the validation succeeds. This is obviously a bug and should be improved by either being able to validate a disabled connector or give the admin some feedback that validation failed because the connector being in a disabled state.

So what can we do to create the connector successfully? Here we have several options. First is to enable the connector during creation and validation and disable it immediately afterwards. Another option is to check if the test email was receive on the remote system and ignore the failed validation. Finally we can test the process in a test (or trial) tenant, however this tenant would likely not be 100% identical to our production tenant.

A service request was raised with Microsoft Support and I'm expecting a fix somewhere in the near feature.

VMware, let's talk about Exchange and best practices

Apologies for the weird spacing, apparently Google changed something so Windows Live Writer can't post to Blogger currently. Posted this via a very weird work-around.

This week I was made aware of some VMware articles and whitepapers about running Microsoft Exchange on VMware VSAN. VMware VSAN is a 'virtual SAN' solution that employs local server storage and presents this to the hypervisor as a pool of shared storage, similar to the approach of Nutanix.

What surprised me, besides the mud slinging between VMware and their partner Nutanix, is the Exchange environment VMware built to execute their performance testing with.

I won't go into the details of the VSAN storage technology in this blog post, nor cover all aspects of the Exchange architecture design process. Today I want to focus on the high-level Exchange architecture used in the whitepapers.

The Exchange 2010 on VSAN whitepaper

First let's take a look at the 2014 whitepaper Microsoft® Exchange Server Performance on VMware Virtual SAN™.

Wait what? Exchange 2010 was used to perform the tests while Exchange 2013 was released almost two years earlier. Exchange 2010 was installed on Windows Server 2008 R2 while Server 2012 was available. Using older software to demonstrate your platforms capabilities may be done for very good reasons, however they were not explained in the whitepaper.

Microsoft introduced server roles in Exchange 2007 and soon discovered the new roles confused the crap out of their customers. So in the Exchange 2010 time-frame the Exchange team started to emphasize why combining the Mailbox, Client Access and Hub Transport roles on a single server was the preferred way to deploy Exchange. Most important reason: less complexity. And with that of course comes a decrease in costs, better reliability and more good things.

There have been situations where multi-role was not the most efficient option. Back when Exchange 2010 was released the most dominant hypervisor vendor was not able to assign more than 4 vCPU to a VM and when they were, they charged extra for that capability. As far as I know none of these limitations apply today so it's unclear why VMware did not deploy Exchange as recommended.

The Exchange 2013 on VSAN 6.0 whitepaper

The updated version of this whitepaper can be found here: Virtualizing Microsoft Applications on VMware Virtual SAN, Reference Architecture. In this whitepaper VMware explains how to deploy a HA Exchange 2013 environment while using the VSAN solution as the storage back-end.

Now starting with Exchange 2013 the Exchange team went a step further and consolidated their recommendations in the Preferred Architecture. This architecture was presented on industry conferences as MEC, TechEd and recently Ignite. In fact, the first article Microsoft released about Exchange 2016 contains a section called Preferred Architecture where the team explains the Exchange 2013 PA remains valid, with some minor updates.

Now let's compared the PA with the architecture VMware describes in this whitepaper.

First of all VMware did not deploy multi-role servers, they deployed 8 Exchange 2013 server where 4 could've done the trick. More servers equals an increase in complexity and costs.

The placement of the "DAG File Cluster" is a bit confusing too, especially because the Exchange Mailbox servers and the FSW are connected by a blue line. I wonder if the author understands the role of the FSW for the cluster.

Database and log files are isolated, this isn't required. And 80 GB is not sufficient for the OS (32 GB), page file (32 GB + 10 MB) and Exchange (30 GB).

VMware uses two network adapters, which nowadays is no longer the best practice.

Not just VMware, Citrix too

Recently I commented on some Exchange whitepapers by Citrix and today I looked at two whitepapers by VMware. Both vendors demonstrate they have limited understanding of Microsoft Exchange and the best practices Microsoft wrote.

Let me be honest. IT has been never more challenging as today. Customer's environments are growing in size and complexity. Trends like virtualization (server, network, storage, application, etcetera), BYOD, consumerization of IT, cloud computing and increasing demands from the business can drive an IT person crazy. This is why it has never been so important for vendors to reduce complexity, make sure recommendations are in line with other vendors and that customers receive all the help they need to implement the solutions in the best way possible.

As an Exchange consultant I often sat with customers discussing Microsoft best practices for Exchange and the storage or virtualization guy showing up with an outdated or slightly inaccurate document instructing the exact opposite. Customers expect the vendors to help them clarify stuff, not to cause even more confusion.

So Citrix, VMware and other vendors too, time to step up your game. Make sure you read and understand every article on the official Exchange Team Blog. Update your whitepapers during their lifetime when progressive insight or updated best practices require it. Ask Microsoft for feedback or involve Subject Matter Experts to review your document and provide some feedback from their perspective. I'm sure that with a little extra effort the quality of the whitepapers can improve a lot.