Outlook Anywhere Kerberos and moving to Exchange 2013? Read this first.

If you run Exchange 2010 and are using Kerberos authentication for Outlook in a load-balanced environment you probably have scheduled the RollAlternateserviceAccountPassword.ps1 script. This script updates the alternate service account credential (ASA credential) and pushes the new value to your CAS servers. Common parameter options are -ToEntireForest, -ToArrayMembers or -CopyFrom with -ToSpecificServers. If you are familiar with the script, I assume they need no clarification.

Now consider a scenario where you used -ToArrayMembers and you add your first Exchange 2013 CAS server to that site. The script uses the Get-ClientAccessArray cmdlet to query the members of this array, this cmdlet returns the Exchange 2013 CAS servers in that site too.

Unfortunately the script is not able to update the ASA credential on both 2010 and 2013 servers. This causes the script to fail when it tries to process the Exchange 2013 CAS server and ultimately cancels the process of updating and synchronizing the ASA credential. Unless you configured specific monitoring for this process you'll probably won't notice the issue before stuff breaks and users start complaining. To check if this issue applies to your environment:

Get-ClientAccessServer -IncludeAlternateServiceAccountCredentialstatus |Fl Name, AlternateServiceAccountConfiguration

By the way, the RollAlternateserviceAccountPassword.ps1 script writes a log file in the $exinstall\Logging\RollAlternateServiceAccountPassword folder. Be ware this log is written on the server where the script is executed, this is not necessarily the server where you scheduled the script to run.

There are several workarounds to prevent this from happening. First you could consider to deploy the Exchange 2013 CAS servers in another site. Another option is to update a single server first and then use the -CopyFrom and -ToSpecificServers switches to update your Exchange 2010 CAS servers which you have to specify.

For more information on planning the migration from Exchange 2010 to Exchange 2013 with regards to Kerberos authentication I recommend this excellent article on the Exchange Team Blog: Exchange 2013 and Exchange 2010 Coexistence with Kerberos Authentication

PowerShell one-liner: Add a computer to a domain or workgroup

Today I ran into an issue where I needed to remove some servers from the domain, perform some tasks and add them again. Adding a server to the domain is very common, use the Add-Computer cmdlet like this:

Add-Computer -DomainName MyDomain.tld

Or even this to change the computer name in the process:

Add-Computer -DomainName MyDomain.tld -NewName server34

To remove a computer from the domain and add it to a workgroup we can use the Remove-Computer cmdlet:

Remove-Computer -Workgroup temp

Or maybe:

Remove-Computer -Workgroup temp -Force -Restart

Makes sense, doesn't it? Don't forget to use an elevated prompt (Run as Administrator) or all you will see is an Access Denied error.

Can't install Edge Transport on domain member server.

I'm in the process of deploying a couple of Edge Transport servers in a DMZ domain. The installation on each of them failed at the same point with the exact same error message:

Active Directory operation failed on localhost. This error is not retriable. Additional information: The parameter is incorrect.
Active directory response: 00000057: LdapErr: DSID-0C090D14, comment: Error in attribute conversion operation, data 0, v23f0

Unfortunately I was not able to find information about the LDAP error code, not even a mention on a website or forum. So I went a couple of steps back and checked my implementation plan. Installing the prerequisites for Edge Transport is fairly simple and the DNS suffix was already set because this server is a domain member. Couldn't think of a mistake I made.

A quick search on the internet pointed met to an article Jaap Wesselius wrote a time ago: Edge Transport server fails in Active Directory domain. The issue Jaap describes is very similar, although the actual LDAP error is slightly different and was present in CU5 and CU6. Unfortunately I can confirm this issue is still there in CU7, assuming both our issues have the same root cause.

Due to time constraints I was not able to involve PSS to investigate the issue further and had to choose for the workaround of installing Exchange while the servers is in Workgroup mode. After the Edge Transport role was installed I was able to add the server to the domain and Exchange was functioning properly.

Concerns about using 3rd party add-ins in Azure

Recently I deployed a WordPress website from the Azure Marketplace. As with most Azure services the process of setting this up is very easy and a lot happens behind the screen.

One of the configuration items is a MySQL database provided by ClearDB.

So after you deploy the solution, the ClearDB database will be linked to your Azure Website. So far so good.

This morning I got a call from the web designer who said 'the server is down'. After some more questions I understood that the website was still available, but the connection with the database failed.

I checked the Azure Health dashboard and all checks were green. Which to be honest did not surprise me because my experience with the availability of Azure services has been very good until today.

Next stop was the Notifications section in the Azure Portal which had nothing too. In the mean time the web designer called and said the database was back but complained he had lost hours of work. At this point I was getting serious doubts about my decision to have my new company website built on Azure infrastructure.

At first I did not notice anything relevant on the ClearDB website until I read the FAQ again and noticed a link to their health information. It was immediately clear they had major issues in my region:

Unfortunately I discovered this hours after the incident happened and there was no mechanism in place for this information to reach me pro-actively.

I had a similar issue when I transferred some resources to a new Azure Subscription and someone deleted my SendGrid subscription. Had to visit the Azure Gallery, purchase the SendGrid service again and change the configuration of my Azure services to use the updated subscribtion.

Both incidents made me realize we still need to think of our services (website, email, ...) as a whole and include all dependent components in the picture. The fact that we are billed by Microsoft and have an manage the service from a single dashboard does not mean we have a single point of contact when it concerns support or health information.

Added to my (long) to-do list: Investigate if I can migrate from ClearDB MySQL to Azure SQL...

Cisco AnyConnect: Failed to initialize connection subsystem

Today I ran into this error when trying to start a Cisco AnyConnect VPN connection: Failed to initialize connection subsystem

After some reading I found the easiest way of fixing this issue is to use the Troubleshoot compaibitlity-wizard against vpnui.exe in C:\Program Files (x86)\Cisco\Cisco AnyConnect Secure Mobility Client.

Click Try recommended settings and note that the wizard has used Windows 8 compatibility mode:

Click Test the program... and apply the settings with Yes, save these settings for this program.

Restart the Cisco AnyConnect Secure Mobility Agent service or reboot your computer.

This happened on Windows 8.1 after I installed some updates, based on reports of other users I suspect Cumulative Security Update for Internet Explorer 11 for Windows 8.1 for x64-based Systems (KB3021952) may be the culprit. Other users report the same issue in Windows 10 Technical Preview.

Cisco AnyConnect observes the IE Work Offline setting for the System Account. Recent updates to IE changes the behaviour of IE which seem to have broken Cisco AnyConnect. Based on my research many similar product use the same approach and can run into issues on Windows 10 TP or older versions of Windows with update KB3021952 installed.

An alternative approach would be to create a key in the registry to fool Cisco AnyConnect. Create a new key GlobalUserOffline with a value of 1 under the path HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings.

Windows Mobile does not support your new SSL certificate

The world is moving away from SHA-1 certificates, which is a good thing from a security perspective. Major vendors like Microsoft and Google are forcing Certificate Authorities to give out certificates with the new SHA-256 hashing algorithm. They do this by simply stop trusting root CA’s which give out SHA-1 certificates after a certain date.

What can you do? Not much actually, because every new SSL certificate you purchase or renew will be SHA-256. Of course there are old operating systems which don't support SHA-256, for instance XP versions older than SP3 and Server 2003 without SP2 installed. So basically every modern OS and even much older versions support SHA-256.

Now for us Exchange guys there may be an exception, this is Windows Mobile 5 and 6 which do not support SHA-256. Yes, this operating system is very old and has been superseded by Windows Phone 7 and newer. However, it has been sold until not very long ago and products with Windows Mobile may be still available. An example of such a device is the Samsung Omnia 735.

So if you have any users with a Windows Mobile device and are planning to renew your Exchange SSL certificate, please be aware of this issue. Depending of your situation you may consider replacing the devices or even move to SHA-1 certificates which are signed by your own CA. This obviously is not a safe long-term solution but it may buy you some time before you can move to SHA-256 certificates.

Fix those silly "106" Performance Counter events on Exchange 2013 servers

You've probably seen them too, lots of errors in the Application log of your Exchange 2013 server from source MSExchange Common and event id 106.

Performance counter updating error. Counter name is PowerShell Average Response Time, category name is MSExchangeRemotePowershell. Optional code: 2. Exception: The exception thrown is : System.InvalidOperationException: The requested Performance Counter is not a custom counter, it has to be initialized as ReadOnly.

The issue is caused by an error in the Exchange setup process where a performance counter definition is tried to read from the wrong location.

The good news is that we can fix this very easy. Copy the following script to a text file and save with the .ps1 extension.

Add-PsSnapin Microsoft.Exchange.Management.PowerShell.Setup
$files = Get-ChildItem $exinstall\setup\perf\*.xml
Write-Host "Registering the perfmon counters"
Write-Host
$count = 0;
foreach ($i in $files)
{
   $count++
   $f =  $i.directory, "\", $i.name -join ""
   Write-Host $count $f -BackgroundColor red
   New-PerfCounters -DefinitionFileName $f
}

Run the script from an Exchange management shell.

If you run into issues you can manually retry the process for that specific performance counter definition. For instance, to retry the failed counter definition from the screenshot above you can retry the action:

Add-PsSnapin Microsoft.Exchange.Management.PowerShell.Setup
New-PerfCounters -DefinitionFileName "C:\Program Files\Microsoft\Exchange Server\V15\setup\perf\WorkerTaskFrameworkPerfCounters.xml"

Interesting detail is that Microsoft apparently wrote a KB article about this issue back in 2013 which I failed to pick up. I modified the script to work on servers with Exchange installed in a non-default path. If you prefer to use the original one, don't forget to change the path manually.

Issue with (and fix for) broken IMAP on Exchange 2013

Recently I ran into an issue configuring some new Exchange 2013 servers for IMAP4. I'll share my configuration, troubleshooting steps and the solution here in this article.

Enabled and started the IMAP4 and IMAP4BE services and created a virtual service on the load balancer listening on port 993. I've done this many times before and was quite surprised that the initial testing with Outlook in IMAP4 mode failed. From a testing perspective I prefer Outlook because it allows not only connecting, but also authentication and actually accessing a mailbox through IMAP4.

I proceeded with a telnet to the load balancer VIP on port 993 and got nothing more than a black screen. Not exactly sure what to expect on port 993 because the encryption I tested port 143 directly against the server, bypassing the load balancer. Here I expected the IMAP banner but saw again just an empty window.

Blank screen, hitting any key resulted in the PowerShell prompt gain. Telnet on the local server to localhost instead of the server name:

So a telnet to localhost results in the IMAP banner, all other options fail. Time to look under the hood and consult Managed Availability. Get-HealthReport reported the Health Set IMAP.Proxy in an offline state:

Setting the state back to active did the trick:

Set-ServerComponentState -Identity ex01 -Component ImapProxy -State Active -Requester HealthAPI

With all components in an active state I was able to connect with IMAP with both telnet and Outlook.

I have seen this issue on multiple servers now, happened to be all running CU7. On the other hand, some other servers did not have this issue. If you run into issues trying to get IMAP4 working on Exchange 2013, maybe the information above helps you to find the cause.

Hyper-V home lab? Deduplication is awesome!

Data deduplication is a Windows feature since Server 2012. Deduplication identifies identical chunks of data, stores a single copy and replaces the remaining copies with a pointer to this copy. Just as with compression, the maximum deduplication rate depends on your data. If every single file is 100% unique and shares no duplicates with other files, then there's not much to deduplicate. However, what if your files are VHDX files and every file contains the same Windows operating system files?

In Server 2012 R2 the Data Deduplication features were improved and deduplication of virtual machine files is now supported. Several limitations apply, for instance this is only supported with VDI workload and the server you enable deduplication on cannot be the Hyper-V server itself. The reason for that is that the actual process of deduplication, which runs as a scheduled task, requires quite a bit of resources and we don't want the performance of the VM's running on the server to be effected.

Microsoft claims that storage savings up to 95% can be achieved.

That's very interesting for business purposes but for my home lab too. My two Hyper-V servers have limited storage capacity and I have to remove unused files now and then to free up disk space. The workload is not VDI and the storage is on the local server so my configuration is not supported. Which is a risk I'm willing to take for my home lab.

A couple of days ago I enabled deduplication on the local data volume of the servers and used the Hyper-V usagetype to enable low-level optimizations for the deduplication of running Hyper-V images. First I had to install the Windows feature:

Add-WindowsFeature FS-Data-Deduplication
Enable-DedupVolume D: -UsageType HyperV

Enabling deduplication added three scheduled tasks under \Microsoft\Windows\Deduplication:

The tasks call ddcpicli.exe with various parameters, the Optimization task runs once a day. Ddcpicli.exe is not meant for manual usage, for that we have Get-DedupJob, Start-DedupJob and Stop-DedupJob.

I was patient and checked the result after some days with Get-DedupStatus:

After reviewing the full output I noticed that deduplication achieved a whopping 49% savings rate, even 52% on my second server!

So bear in mind, unless you're deduplicating VDI VM files on a remote Server 2012 R2 fileserver you're unsupported. If that's not a problem for your lab, try it for yourself. Before you do, make sure you've read the following articles:

What's New in Data Deduplication in Windows Server

Deploying Data Deduplication for VDI storage in Windows Server 2012 R2

PowerShell one-liner: Start a service on a remote server

A while back I wrote an article on how to start a service on a remote computer. Because Start-Service does not have a -ComputerName parameter I suggested this alternative approach:

(Get-WmiObject -Computer server1 Win32_Service -Filter "Name='msExchangePOP3'").InvokeMethod("StartService",$null)

While this works perfectly, the syntax is a bit hard to remember. Gladly I became aware of a much simpler way to do this. While we can't use Start-Service, we can use Set-Service to set the service to the Running status.

Set-Service msExchangePOP3 -Status running –ComputerName server1

That's cool, isn't it? So for instance when you're in the process of enabling IMAP on an Exchange 2013 server, your script could be something like this:

Set-Service msExchangeIMAP4 -StartupType Automatic –ComputerName server1
Set-Service msExchangeIMAP4 -Status Running –ComputerName server1
Set-Service msExchangeIMAP4BE -StartupType Automatic  –ComputerName server1
Set-Service msExchangeIMAP4BE -Status Running –ComputerName server1

PowerShell one-liner: Reboot a remote server

To reboot a remote server we can use the Restart-Computer cmdlet, use the -ComputerName parameter to tell PowerShell to execute the command against a remote server. To restart server Host01:

Restart-Computer -ComputerName host01

Just like rebooting a local server PowerShell will fail if another user is still logged on to the server. Add the -Force parameter to force the reboot.

Exchange 2013: Failed to detect the bitlocker state for EDS log drive...

On most, if not all, Exchange 2013 CU7 servers the following error appears in the Application Log:

Log Name:      Application
Source:        MSExchangeDiagnostics
Date:          2-2-2015 08:51:51
Event ID:      1039
Task Category: General
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      server.fqdn
Description:
Failed to detect the bitlocker state for EDS log drive 'D:\'.
Exception
System.Management.ManagementException: Invalid namespace
   at System.Management.ManagementException.ThrowWithExtendedInfo(ManagementStatus errorCode)
   at System.Management.ManagementScope.InitializeGuts(Object o)
   at System.Management.ManagementScope.Initialize()
   at System.Management.ManagementObjectSearcher.Initialize()
   at System.Management.ManagementObjectSearcher.Get()
   at Microsoft.Exchange.Diagnostics.Service.DiagnosticsService.DriveLocked(String diagnosticsRootDrive)

Apparently the Exchange Diagnostics Service (EDS) tries to determine if Bitlocker is enabled on the volume where the Diagnostics log files are being stored.

I have seen this issue on all Exchange 2013 CU7 servers both in my lab and production environments, as well have other people according this TechNet forums discussion. At time the Error event does not seem to be an actual issue, my guess is that it be safely ignored. Hopefully it will be gone in the next CU...

Fixing the MSExchangeApplicationLogic event id 3018 error

On an Exchange 2013 server you can run into the following error:

Log Name:      Application
Source:        MSExchangeApplicationLogic
Date:          2-2-2015 08:57:47
Event ID:      3018
Task Category: Extension
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      server.fqdn
Description:
Scenario[ServiceHealth]: GetConfig. CorrelationId: 2176756c-f261-4085-a812-b5e4bda31f0d. The request failed. Mailbox:  Url: https://o15.officeredir.microsoft.com/r/rlidMktplcWSConfig15?CV=15.0.1044.25&Client=WAC_Outlook&corr=2176756c-f261-4085-a812-b5e4bda31f0d Exception: System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 23.97.152.190:443

This error occurs because Exchange tries to download an XML file which fails, usually because the server has no internet access. The error can be prevented by having Exchange make this call through a proxy server, use the Set-ExchangeServer cmdlet to achieve this:

Set-ExchangeServer Server01 -InternetWebProxy http://10.3.3.3:80