KEMP LoadMaster and Exchange 2013: Check your Server Check configuration

If you deploy a KEMP LoadMaster to load balance Exchange (which you should!) you may see an unusual behavior where the LoadMaster treats a service failure incorrectly as a server failure. First let me explain a very typical configuration example to demonstrate the issue, after that I'll explain how to fix.

I won't go into the details of deploying the LoadMaster or Exchange 2013, if you're reading this article you're supposed to have a good understanding of Exchange 2013 load balancing and the basics of working with the KEMP LoadMaster.

I start my configuration by downloading the latest Exchange 2013 Templates from the KEMP Technologies website. In this example I used Core services: MAPI, SMTP and Unified HTTP/HTTPS because I'm not going to enable ESP for this service.

So a new Virtual Service is created with the Exchange 2013 HTTPS Reencrypted template.

Next step is to assign a SSL certificate to the Virtual Service:

And add the real servers to all nine sub-Virtual Services:

The result is a nice and healthy Virtual Service:

So far so good? Well, almost… Let's see what happens when one of our Real Servers encounters an issue. To do so we simulate an unhealthy OWA, resulting in having Managed Availability no longer reporting a 200 OK when the /owa/healthcheck.htm url is queried.

Set-ServerComponentState ex01 -State inactive -Component owaproxy -Requester healthapi

Now if we check the health of the Virtual Service in the KEMP WUI we expect it to report an unhealthy Real Server for the OWA sub-Virtual Service. Instead it displays a failed RS for all services:

In the Warning Log is an endless series of these error messages:

Mar 30 13:56:04 lb100 l4d: Removing RS 192.168.200.182:443 from VS 192.168.200.200:443(E2013HTTPS) - EOF or Incorrect data received
Mar 30 13:56:04 lb100 last message repeated 5 times
Mar 30 13:56:13 lb100 l4d: Adding RS 192.168.200.182:443 to VS 192.168.200.200:443(E2013HTTPS)
Mar 30 13:56:13 lb100 last message repeated 5 times
Mar 30 13:56:13 lb100 l4d: Removing RS 192.168.200.182:443 from VS 192.168.200.200:443(E2013HTTPS) - EOF or Incorrect data received
Mar 30 13:56:13 lb100 last message repeated 5 times
Mar 30 13:56:22 lb100 l4d: Adding RS 192.168.200.182:443 to VS 192.168.200.200:443(E2013HTTPS)
Mar 30 13:56:22 lb100 last message repeated 5 times

Apparently the LoadMaster detects the entire RS unavailable and removes the RS from the VS. Now typically we have enabled the Drop Connections on RS failure feature because this is something you want for load balancing Exchange. The result is that your Outlook uses will be disconnected and forced to reconnect every time the LoadMaster removes 'their' RS from the VS. Especially for Outlook in online mode this will result in helpdesk calls and unhappy users.

I worked with KEMP Support to troubleshoot this unexpected behavior and the root cause was found pretty fast. By default the Real Server Check uses HTTP/1.1 to query the healthcheck.htm url, as can be seen here:
HTTP/1.1 is a bit more efficient than the default of HTTP/1.0 because it bundles multiple requests. Unfortunately this breaks our per-service health checks because the LoadMaster is no longer able to detect which subVS was the unhealthy one, as the result of that the entire RS is removed from the service.

My recommendation is to disable the Use HTTP/1.1 feature of all subVS to restore normal behavior.

KEMP Support, as always, was great to assist us with this issue. I left a Feature Request to ask them to update the Exchange Templates to remove the HTTP/1.1 checkbox by default.

Citrix NetScaler configuration notes for Exchange 2013

So writing an improved Citrix NetScaler deployment guide for Exchange 2013 is on my to-do list for a long time now, and to be honest I don't think I'm able to dedicate the time needed for this project. So I'll leave my notes from a similar deployment I recently carried out in a lab environment.

Disclaimer: This is work in process and is not meant as a replacement for the Citrix documentation. Maybe someone can use this for another project or as an example of how to document the configuration of a NetScaler for load balancing Exchange 2013.

Part A: Create a CS Virtual Server

1. Content Switching Virtual Servers
add cs vserver vserver-cs-exchange-https SSL 123.123.123.123 443

Part B: Create a Load Balancing setup

2. Load Balancing Virtual Servers
add lb vserver vserver-lb-exchange-owa ssl 10.0.0.1 443
add lb vserver vserver-lb-exchange-ecp ssl 10.0.0.2 443
add lb vserver vserver-lb-exchange-ews ssl 10.0.0.3 443
add lb vserver vserver-lb-exchange-oab ssl 10.0.0.4 443
add lb vserver vserver-lb-exchange-rpc ssl 10.0.0.5 443
add lb vserver vserver-lb-exchange-eas ssl 10.0.0.6 443
add lb vserver vserver-lb-exchange-aut ssl 10.0.0.7 443

3. Service Groups
add servicegroup servicegroup-exchange-https SSL

4. Bind service to servicegroup
bind servicegroup servicegroup-exchange-https 192.168.101.45 443
bind servicegroup servicegroup-exchange-https 192.168.101.46 443

5. Bind Service Groups to LB Virtual Servers
bind lb vserver vserver-lb-exchange-owa servicegroup-exchange-https
bind lb vserver vserver-lb-exchange-ecp servicegroup-exchange-https
bind lb vserver vserver-lb-exchange-ews servicegroup-exchange-https
bind lb vserver vserver-lb-exchange-oab servicegroup-exchange-https
bind lb vserver vserver-lb-exchange-rpc servicegroup-exchange-https
bind lb vserver vserver-lb-exchange-eas servicegroup-exchange-https
bind lb vserver vserver-lb-exchange-aut servicegroup-exchange-https

6. Bind certificate and key to CS and LB Virtual Servers
bind ssl vserver vserver-cs-exchange-https -certkeyName wildcard
bind ssl vserver vserver-lb-exchange-owa -certkeyName wildcard
bind ssl vserver vserver-lb-exchange-ecp -certkeyName wildcard
bind ssl vserver vserver-lb-exchange-ews -certkeyName wildcard
bind ssl vserver vserver-lb-exchange-oab -certkeyName wildcard
bind ssl vserver vserver-lb-exchange-rpc -certkeyName wildcard
bind ssl vserver vserver-lb-exchange-eas -certkeyName wildcard
bind ssl vserver vserver-lb-exchange-aut -certkeyName wildcard

7. Monitors
add lb mon monitor-exchange-owa HTTP-ECV -interval 30 -secure YES -send "GET /owa/healthcheck.htm" -recv "200 OK"
add lb mon monitor-exchange-ecp HTTP-ECV -interval 30 -secure YES -send "GET /ecp/healthcheck.htm" -recv "200 OK"
add lb mon monitor-exchange-ews HTTP-ECV -interval 30 -secure YES -send "GET /ews/healthcheck.htm" -recv "200 OK"
add lb mon monitor-exchange-oab HTTP-ECV -interval 30 -secure YES -send "GET /oab/healthcheck.htm" -recv "200 OK"
add lb mon monitor-exchange-rpc HTTP-ECV -interval 30 -secure YES -send "GET /rpc/healthcheck.htm" -recv "200 OK"
add lb mon monitor-exchange-eas HTTP-ECV -interval 30 -secure YES -send "GET /microsoft-server-activesync/healthcheck.htm" -recv "200 OK"
add lb mon monitor-exchange-aut HTTP-ECV -interval 30 -secure YES -send "GET /autodiscover/healthcheck.htm" -recv "200 OK"

8. Bind monitor to Service Group
bind lb monitor monitor-exchange-owa servicegroup-exchange-https
bind lb monitor monitor-exchange-ecp servicegroup-exchange-https
bind lb monitor monitor-exchange-ews servicegroup-exchange-https
bind lb monitor monitor-exchange-oab servicegroup-exchange-https
bind lb monitor monitor-exchange-rpc servicegroup-exchange-https
bind lb monitor monitor-exchange-eas servicegroup-exchange-https
bind lb monitor monitor-exchange-aut servicegroup-exchange-https

9. Content Switching policies
add cs policy pol-exchange-owa -url "/owa/*"
add cs policy pol-exchange-ecp -url "/ecp/*"
add cs policy pol-exchange-ews -url "/ews/*"
add cs policy pol-exchange-oab -url "/oab/*"
add cs policy pol-exchange-rpc -url "/rpc/*"
add cs policy pol-exchange-eas -url "/microsoft-server-activesync/*"
add cs policy pol-exchange-aut -url "/autodiscover/*"

10. Bind CS policy to Virtual Server
bind cs vserver vserver-cs-exchange-https -lbvserver vserver-lb-exchange-owa
bind cs vserver vserver-cs-exchange-https vserver-lb-exchange-owa -policyName pol-exchange-owa
bind cs vserver vserver-cs-exchange-https vserver-lb-exchange-ecp -policyName pol-exchange-ecp
bind cs vserver vserver-cs-exchange-https vserver-lb-exchange-ews -policyName pol-exchange-ews
bind cs vserver vserver-cs-exchange-https vserver-lb-exchange-oab -policyName pol-exchange-oab
bind cs vserver vserver-cs-exchange-https vserver-lb-exchange-rpc -policyName pol-exchange-rpc
bind cs vserver vserver-cs-exchange-https vserver-lb-exchange-eas -policyName pol-exchange-eas
bind cs vserver vserver-cs-exchange-https vserver-lb-exchange-aut -policyName pol-exchange-aut

Citrix and Microsoft guides, what's up with that?

Citrix recently announced an Implementation Guide for Microsoft Office 365 for Citrix XenApp and XenDesktop 7.x. After reading their recent deployment guides I noticed several mistakes which makes very clear to me that the writers may understand Citrix very well but not the Microsoft technologies and products. Now the purpose of these documents is to make sure that our customers and partners understand the technologies well enough to guarantee a successful implementation. I really wonder why they don't have a Subject Matter Expert proofreading the documents before they get released. Let me give a few examples.

The latest document starts with an error in the first paragraph:

Office 365 ProPlus includes a combination of online-based applications (Outlook, Word, Excel, PowerPoint and OneNote) that are accessed from anywhere via a web browser, as well as the latest traditional, locally installed version of Microsoft Office.

No it's not, Office ProPlus is the locally installed version of Microsoft Office.

Included with Office 365 ProPlus is an online email account with 50GB of storage and 1TB of file storage per user with OneDrive for Business.

No it's not, Office 365 ProPlus is not the name of one of the Office 365 subscriptions, it's the version of Microsoft Office you install on your computer.

Another example is Deploying Citrix NetScaler with Microsoft Exchange 2013 for GSLB.

Per service health checks, single unbound namespace, stretched DAG, nice! But why is there a CAS server in the DAG? Or more important, why is the DAG load balanced? When talking about Exchange high availability it's key to have a good understanding of the difference between Client Access HA and Mailbox data HA. The traffic to the Client Access servers is load balanced. Mailbox servers in a DAG enable the organization to deploy multiple copies of the database and store them on another DAG member server. The Client Access servers proxy the user's request to a Mailbox server.

Different clients add different suffixes to the domain name when they connect to virtual server. For example, /owa for web browser clients, /oa for Outlook anywhere.

That's the /rpc virtual directory, not /oa.

NetScaler GSLB also enables to maintain availability in case of site level disaster in which one of the sites is completely unavailable. This is shown in Figure 4. When there is hot sync between the mailbox servers and the user information is available on all the mailbox servers, then all the requests of site 1 can be completely served from site 2.

What is hot sync between the mailbox servers? I suppose the writer means that the database has a replica or copy on a server in the second site.

And then there is the Guide to Deploying Microsoft Exchange 2013 with Citrix NetScaler.

A snippet from a bit complex drawing. All servers, including the domain controllers are multi-homed. This is at least very uncommon and certainly not according best practices. And again there's a (single?) CAS server in the DAG that does not belong there. It should've been at least two CAS servers to demonstrate load balancing of the CAS servers.

And then there's the split-role/multi-role debate, where Microsoft recommends to deploy multi-role servers since the 2010 timeframe and still does for 2013. I guess the writers tried to update the document by writing "Exchange 2013 Multirole Servers MB/CAS" under the picture but forgot to actually update the diagram.

Is not going to work, oa needs to be rpc.

In the configuration shown above, a single namespace is used for all Exchange protocols. For example, for web access, the namespace is https://mail.ctxns.net/owa, and for Outlook clients the namespace is https://mail.ctxns.net/oa.

It's still /rpc, not /oa. And where is /Autodiscover, did we forgot this one?

More important flaw of the document is that the first half is the copy and past of some TechNet articles and some slides Microsoft presented at a tech conference, the seconds halve is just a series of screenshots. For me as an Exchange admin it does not really explain an good overview of load balancing with NetScaler, it does not dive in to the details as for instance session time-out settings, why we use least connection and if we can configure a slow start time to prevent an overloaded server after a reboot, transparent or non-transparent load balancing, routing considerations, etc.

And from a load balancer perspective I'm not sure how well the NetScaler guy understands load balancing an Exchange 2013 solution after reading the deployment guide. Especially with the small mistakes and bad representation of the architecture (mainly in the GSLB document).

Let's wrap-up my complaints with a positive remark. The latest version of the document is an improvement over an older version which had a dedicated set of Client Access servers in the perimeter network. I'm being sarcastic of course, I think there's room left for improvement. I mean the NetScaler is an interesting product, especially for customers looking for a CAG replacement. But when it comes to documentation and ease of configuration other vendors as KEMP Technologies have a better story. Personally I prefer an affordable load balancer with excellent documentation by a company who shows they understand their work loads over a powerful network appliance which is too complex to configure correctly and lacks proper documentation. This is where Citrix can learn from KEMP technologies.

Health Manager service not starting on Exchange 2013 Edge Transport servers

This is one of those issues in that was in Exchange 2013 RTM is is still there in CU8. I'm pretty sure it's very easy to fix and Microsoft PSS is aware of the issue, but anyway this is the error I'm talking about:

The Microsoft Exchange Health Manager service depends on the following service: MSExchangeADTopology. This service might not be installed.

This of course makes perfect sense because the Microsoft Exchange Active Directory Topology Service is not installed on an Edge Transport server. So in order to remove the dependency on the MSExchangeADTopology service we need to fire up Regedit and navigate to the following key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeHM. Open the Value DependOnService and remove MSExchangeADTopology from the Value Data.

Make sure the only remaining entry is eventlog and click OK to save. Reboot your server and verify that the Microsoft Exchange Health Manager is running.

Need to disable SSL3.0 for Exchange 2013 SMTP? Install CU8 to make it work

With the recent security issues with SSL 3.0 many organizations are in the process of disabling this vulnerable protocol for Exchange 2013 servers. Unfortunately SMTP appeared to keep using SSL 3.0 even after an administrator configured the server to use the more secure TLS 1.1 or 1.2 protocol.

This issue has been fixed in Exchange 2013 CU8. More information (a little bit) in the following KB article: SMTP is not transported over TLS 1.1 or TLS 1.2 protocol in an Exchange Server 2013 environment

Store Outlook signatures in the cloud or in the mailbox. Julia White: "It's on the list"

This has been a feature request since Outlook 97 or maybe even Exchange client. Every Exchange customer wants to be able to store and manage the email signature in a way that is consistent in all clients, whether it's Outlook, OWA or even ActiveSync devices. AutoDiscover makes setting up your device to connect to your email very easy, then why is the e-mail signature a client-side feature and is it not stored in the mailbox where it belongs just like message rules and the OOF configuration?

Data Experience Expert (don't know what that is, tried to read the explanation but still don't understand) Chris Woodruff Tweeted earlier today:

I realy wish that Office suite would store things like email signatures in the cloud

— Chris Woodruff (@cwoodruff) March 12, 2015

Many people agreed, amongst them was Product Manager for Microsoft's Office Division Julia White.

@devinganger @marypcbuk @cwoodruff totally agree! On the list...will see if we can pull up sooner...

— Julia White (@julwhite) March 12, 2015

I find this promising. Microsoft has shown to invest more than ever in the Office products (including Exchange, SharePoint and Lync/Skype for Business) these days, driven by the enormous push to make Office 365 a success. let's hope this feature gets the attention it deserves.

If you agree, please show your support by voting on this suggestion on the Microsoft Exchange Improvement Suggestions website: Outlook signatures saved in Mailbox

PowerShell one-liner: How to find the AD site name?

How can we query the AD site name of a server with PowerShell? One way to do this is through the .NET ActiveDirectorySite Class:

[System.DirectoryServices.ActiveDirectory.ActiveDirectorySite]::GetComputerSite().Name

An alterative approach is to query the value for the DynamicSiteName registry key under HKLM:\SYSTEM\CurrentControlSet\services\Netlogon\Parameters.

(Get-ItemProperty "HKLM:\System\CurrentControlSet\Services\Netlogon\parameters").DynamicSiteName

Exchange and support for CNG/KSP certificates

Cryptography Next Generation (CNG) is a set of APIs which can be found in Windows Server 2008 and newer. CNG offers an advanced set of features to create hashes, encrypt and decrypt data and to manage keys and cryptographic providers. CNG implements the United States government's Suite B cryptographic algorithms. As the hipsters say, CNG is the new PKI but you probably haven't heard of it.

One of the features of CNG is to create certificates that that use a Key Storage Provider (KSP) to store the private key, as opposed to a Cryptographic Service Provider (CSP) like regular certificates do. Exchange does not support these types of certificates for securing OWA and ECP. You will notice this immediately because your users will return back in the FBA screen after logging in after you installed such a certificate.

Read the following article for more information and workarounds: Outlook Web App and ECP redirect to the FBA page in Exchange Server 2013

Free KEMP LoadMaster load balancer!

This is very cool! KEMP gives away the LoadMaster Application Delivery Controller for free. The LoadMaster for Azure was already free, now the virtual appliance is available in a free edition too. Available for all supported hypervisors (VMware, Hyper-V, KVM, Xen, Oracle VirtualBox).

The free VLM has some limitations, for instance the HA setup with an active and hot stand-by unit is not supported. Another important limitation is that the free LoadMaster doesn't come with the awesome support paying customers receive. Also there are some bandwidth and SSL TPS limitations, all in all not much special for most home, lab, testing and other non-production deployments.

Get yours now at http://freeloadbalancer.com/