Wednesday, May 20, 2015

Did you know there used to be a KB Article called "How to ask a question"?

If you spend some time in online communities you may have seen people asking questions without making any effort. Some people fail to choose a descriptive title or expect they get more attention with “HELP MY SERVER IS BROKEN, URGENT!!!”. Others forget to do an online search for information first or omit the actual error message or troubleshooting they already did before asking for help.

Back in the early 2000’s when I answered questions in the Microsoft newsgroups I often pointed people to this Knowledge Base article: How to ask a question. The article was written by Daniel Petri, a former Exchange and current Directory Services MVP and maybe best known as the founder of the Petri IT knowledgebase. The article was structured just as any other articles including sections Summary, Symptoms, Cause and Resolution. The tone was polite, respectful but for some reason I found it very satisfying to answer such a bad question with a link to https://support.microsoft.com/?id=555375.

image

I was a bit disappointed to discover that Microsoft replaced the original KB555375 with a more political correct article. But of course you can still access the original version through web.archive.org: How to ask a question.

Tuesday, May 19, 2015

PAL tool now works with Exchange 2013!

PAL or Performance Analyzer for Logs is a handy tool to assist with performance monitoring and troubleshooting. Many Exchange admins have used this tool to quickly create a performance counter set and scan the results against a threshold file. PAL outputs a report with easy graphs and performance alerts. This report is a great help to get a quick overview of the performance of a server and indication of possible performance bottlenecks.

Unfortunately there was no threshold file for Exchange 2013, partly because Microsoft never published the detailed performance counter information as they did for 2007 and 2010. I am very excited that Adrian Moore (admoore@microsoft.com) found the time to write the threshold file for Exchange 2013, which is included with PAL version 2.7.3.

Download PAL and read more at the PAL site on Codeplex.

Tuesday, May 12, 2015

Allow or disallow Outlook Online Mode per user

Outlook Cached Mode offloads IOPs from the server to the workstation, provides offline access and also makes the user experience less vulnerable to unreliable or high-latency internet connections. So Cached Mode is recommended for Exchange Online and is the the preferred mode to connect Outlook with Exchange.

Unfortunately there are some scenario's where you can't enable Cached Mode, for instance stateless (VDI) desktops or RDS/TS hosts when customers for some reason are not able to store the .OST file on a fileserver.

So far nothing new for you as an experience Exchange admin. But did you know that you can specify per-mailbox if it's allowed to access with Outlook in Online Mode? This can be controlled with the Set-CASMailbox cmdlet. This cmdlet is available in both Exchange 2010 and 2013, however the switch to allow Online Mode is not available in Exchange Online.

It's very easy to block access with Outlook in Online Mode for a single mailbox:

Get-Mailbox <user> | Set-CASMailbox -MAPIBlockOutlookNonCachedMode:$true

image

A value of True enables the blocking of Online Mode clients, set the value to False to disable checking of the connection mode. False is the default value.

The user experience is not that great, a rather generic error message is displayed when the user tries to connect in Online Mode:

image

So if you have the requirement to block Outlook in Online Mode, you can do this with the Set-CASMailbox and the MAPIBlockOutlookNonCachedMode switch.

Beware though, setting MAPIBlockOutlookNonCachedMode to $true breaks New-MailboxExportRequest which means you can't export the content from this mailboxes to PST. Not something you would do every day, but good to know.

Monday, May 11, 2015

Microsoft Ignite 2015: Things can only get better

Last week I was lucky enough to be able to attend Microsoft's new flagship conference Microsoft Ignite 2015. Ignite was meant to replace TechEd, MMS, MEC and a bunch of other Microsoft tech conferences. MEC 2014 was absolutely awesome because of the content of the sessions, the relatively small scale of the conference and venue and most important: ample opportunity for interactions with speakers, product group members, MVPs, MCMs and others peers from the Exchange or Office 365 field.

From the initial announcement it was clear that Microsoft had to work very very hard to offer a similar experience. The short answer: they failed to deliver. I won't go into the details too much, read one of these articles to get a better understanding of my experience:

As you may have noticed the feedback is very consistent and I agree with most of it. For me personally the quality and availability of food and drinks was the least of my concerns. It was the immense scale of the conference (23.000 attendees!) that made it very hard to do anything else then wait in lines or try to reach the next session room in 30 minutes and be able to enter before it's full.

Line for lunch starting in the bridge from the South Building

And being there with almost 23.000 attendees also means it was very rare to meet people, let alone to have a chat about the session contents. For me this was the greatest value of MEC and the greatest let down of Ignite.

Despite the many challenges I had a great time though. After a bit of a slow start, the Exchange track delivered some awesome sessions. Meet Exchange Server 2016 was a bit disappointing for a more infrastructure focused person like me, but the next sessions delivered great content, especially Exchange Server Preferred Architecture and Deploying Exchange Server 2016. And it was great to meet some people I only knew from their Twitter handle until then.

Ignite 2016 has been announced and will be held in the same city, same venue. So the distance between the venues and the hotels will remain the same, but I sure hope Microsoft can fix many of the issues with the first Ignite. Let's look at it this way: Ignite can 2016 can only get better!

Tuesday, April 14, 2015

Exchange 2010 SP3: UCMA requires the following missing Windows features

When installing Exchange 2010 SP3 on a server with Windows Server 2012 you need to install some prerequisites first. Each server with the Client Access role requires the Unified Communications Managed API 4.0 Runtime.

You may run into this error message when you try to install:

image

Microsoft Unified Communications Managed API 4.0, Runtime requires the following missing Windows features.

  • Media Foundation

This error occurs when you followed the instructions on this page: Exchange 2010 Prerequisites (Install the Windows Server 2012 operating system prerequisites). For a server with the three basic roles Microsoft says you need to install the following operating system components:

image

Unfortunately Server-Media-Foundation is missing, that's why the UCMA installation fails. So we need to add this one too:

Add-WindowsFeature Server-Media-Foundation

And while we're on it, don't forget to add Telnet-Client because this basic troubleshooting tool should be installed on every server with Exchange by default.

Monday, March 30, 2015

KEMP LoadMaster and Exchange 2013: Check your Server Check configuration

If you deploy a KEMP LoadMaster to load balance Exchange (which you should!) you may see an unusual behavior where the LoadMaster treats a service failure incorrectly as a server failure. First let me explain a very typical configuration example to demonstrate the issue, after that I'll explain how to fix.

I won't go into the details of deploying the LoadMaster or Exchange 2013, if you're reading this article you're supposed to have a good understanding of Exchange 2013 load balancing and the basics of working with the KEMP LoadMaster.

I start my configuration by downloading the latest Exchange 2013 Templates from the KEMP Technologies website. In this example I used Core services: MAPI, SMTP and Unified HTTP/HTTPS because I'm not going to enable ESP for this service.

So a new Virtual Service is created with the Exchange 2013 HTTPS Reencrypted template.

image

Next step is to assign a SSL certificate to the Virtual Service:

image

And add the real servers to all nine sub-Virtual Services:

image

The result is a nice and healthy Virtual Service:

image

So far so good? Well, almost… Let's see what happens when one of our Real Servers encounters an issue. To do so we simulate an unhealthy OWA, resulting in having Managed Availability no longer reporting a 200 OK when the /owa/healthcheck.htm url is queried.

Set-ServerComponentState ex01 -State inactive -Component owaproxy -Requester healthapi

image

Now if we check the health of the Virtual Service in the KEMP WUI we expect it to report an unhealthy Real Server for the OWA sub-Virtual Service. Instead it displays a failed RS for all services:

image

In the Warning Log is an endless series of these error messages:

Mar 30 13:56:04 lb100 l4d: Removing RS 192.168.200.182:443 from VS 192.168.200.200:443(E2013HTTPS) - EOF or Incorrect data received
Mar 30 13:56:04 lb100 last message repeated 5 times
Mar 30 13:56:13 lb100 l4d: Adding RS 192.168.200.182:443 to VS 192.168.200.200:443(E2013HTTPS)
Mar 30 13:56:13 lb100 last message repeated 5 times
Mar 30 13:56:13 lb100 l4d: Removing RS 192.168.200.182:443 from VS 192.168.200.200:443(E2013HTTPS) - EOF or Incorrect data received
Mar 30 13:56:13 lb100 last message repeated 5 times
Mar 30 13:56:22 lb100 l4d: Adding RS 192.168.200.182:443 to VS 192.168.200.200:443(E2013HTTPS)
Mar 30 13:56:22 lb100 last message repeated 5 times

imageApparently the LoadMaster detects the entire RS unavailable and removes the RS from the VS. Now typically we have enabled the Drop Connections on RS failure feature because this is something you want for load balancing Exchange. The result is that your Outlook uses will be disconnected and forced to reconnect every time the LoadMaster removes 'their' RS from the VS. Especially for Outlook in online mode this will result in helpdesk calls and unhappy users.


I worked with KEMP Support to troubleshoot this unexpected behavior and the root cause was found pretty fast. By default the Real Server Check uses HTTP/1.1 to query the healthcheck.htm url, as can be seen here:
image
HTTP/1.1 is a bit more efficient than the default of HTTP/1.0 because it bundles multiple requests. Unfortunately this breaks our per-service health checks because the LoadMaster is no longer able to detect which subVS was the unhealthy one, as the result of that the entire RS is removed from the service.


My recommendation is to disable the Use HTTP/1.1 feature of all subVS to restore normal behavior.


KEMP Support, as always, was great to assist us with this issue. I left a Feature Request to ask them to update the Exchange Templates to remove the HTTP/1.1 checkbox by default.

Sunday, March 22, 2015

Citrix NetScaler configuration notes for Exchange 2013

So writing an improved Citrix NetScaler deployment guide for Exchange 2013 is on my to-do list for a long time now, and to be honest I don't think I'm able to dedicate the time needed for this project. So I'll leave my notes from a similar deployment I recently carried out in a lab environment.

Disclaimer: This is work in process and is not meant as a replacement for the Citrix documentation. Maybe someone can use this for another project or as an example of how to document the configuration of a NetScaler for load balancing Exchange 2013.

Part A: Create a CS Virtual Server

1. Content Switching Virtual Servers
add cs vserver vserver-cs-exchange-https SSL 123.123.123.123 443

Part B: Create a Load Balancing setup

2. Load Balancing Virtual Servers
add lb vserver vserver-lb-exchange-owa ssl 10.0.0.1 443
add lb vserver vserver-lb-exchange-ecp ssl 10.0.0.2 443
add lb vserver vserver-lb-exchange-ews ssl 10.0.0.3 443
add lb vserver vserver-lb-exchange-oab ssl 10.0.0.4 443
add lb vserver vserver-lb-exchange-rpc ssl 10.0.0.5 443
add lb vserver vserver-lb-exchange-eas ssl 10.0.0.6 443
add lb vserver vserver-lb-exchange-aut ssl 10.0.0.7 443

3. Service Groups
add servicegroup servicegroup-exchange-https SSL

4. Bind service to servicegroup
bind servicegroup servicegroup-exchange-https 192.168.101.45 443
bind servicegroup servicegroup-exchange-https 192.168.101.46 443

5. Bind Service Groups to LB Virtual Servers
bind lb vserver vserver-lb-exchange-owa servicegroup-exchange-https
bind lb vserver vserver-lb-exchange-ecp servicegroup-exchange-https
bind lb vserver vserver-lb-exchange-ews servicegroup-exchange-https
bind lb vserver vserver-lb-exchange-oab servicegroup-exchange-https
bind lb vserver vserver-lb-exchange-rpc servicegroup-exchange-https
bind lb vserver vserver-lb-exchange-eas servicegroup-exchange-https
bind lb vserver vserver-lb-exchange-aut servicegroup-exchange-https

6. Bind certificate and key to CS and LB Virtual Servers
bind ssl vserver vserver-cs-exchange-https -certkeyName wildcard
bind ssl vserver vserver-lb-exchange-owa -certkeyName wildcard
bind ssl vserver vserver-lb-exchange-ecp -certkeyName wildcard
bind ssl vserver vserver-lb-exchange-ews -certkeyName wildcard
bind ssl vserver vserver-lb-exchange-oab -certkeyName wildcard
bind ssl vserver vserver-lb-exchange-rpc -certkeyName wildcard
bind ssl vserver vserver-lb-exchange-eas -certkeyName wildcard
bind ssl vserver vserver-lb-exchange-aut -certkeyName wildcard

7. Monitors
add lb mon monitor-exchange-owa HTTP-ECV -interval 30 -secure YES -send "GET /owa/healthcheck.htm" -recv "200 OK"
add lb mon monitor-exchange-ecp HTTP-ECV -interval 30 -secure YES -send "GET /ecp/healthcheck.htm" -recv "200 OK"
add lb mon monitor-exchange-ews HTTP-ECV -interval 30 -secure YES -send "GET /ews/healthcheck.htm" -recv "200 OK"
add lb mon monitor-exchange-oab HTTP-ECV -interval 30 -secure YES -send "GET /oab/healthcheck.htm" -recv "200 OK"
add lb mon monitor-exchange-rpc HTTP-ECV -interval 30 -secure YES -send "GET /rpc/healthcheck.htm" -recv "200 OK"
add lb mon monitor-exchange-eas HTTP-ECV -interval 30 -secure YES -send "GET /microsoft-server-activesync/healthcheck.htm" -recv "200 OK"
add lb mon monitor-exchange-aut HTTP-ECV -interval 30 -secure YES -send "GET /autodiscover/healthcheck.htm" -recv "200 OK"

8. Bind monitor to Service Group
bind lb monitor monitor-exchange-owa servicegroup-exchange-https
bind lb monitor monitor-exchange-ecp servicegroup-exchange-https
bind lb monitor monitor-exchange-ews servicegroup-exchange-https
bind lb monitor monitor-exchange-oab servicegroup-exchange-https
bind lb monitor monitor-exchange-rpc servicegroup-exchange-https
bind lb monitor monitor-exchange-eas servicegroup-exchange-https
bind lb monitor monitor-exchange-aut servicegroup-exchange-https

9. Content Switching policies
add cs policy pol-exchange-owa -url "/owa/*"
add cs policy pol-exchange-ecp -url "/ecp/*"
add cs policy pol-exchange-ews -url "/ews/*"
add cs policy pol-exchange-oab -url "/oab/*"
add cs policy pol-exchange-rpc -url "/rpc/*"
add cs policy pol-exchange-eas -url "/microsoft-server-activesync/*"
add cs policy pol-exchange-aut -url "/autodiscover/*"

10. Bind CS policy to Virtual Server
bind cs vserver vserver-cs-exchange-https -lbvserver vserver-lb-exchange-owa
bind cs vserver vserver-cs-exchange-https vserver-lb-exchange-owa -policyName pol-exchange-owa
bind cs vserver vserver-cs-exchange-https vserver-lb-exchange-ecp -policyName pol-exchange-ecp
bind cs vserver vserver-cs-exchange-https vserver-lb-exchange-ews -policyName pol-exchange-ews
bind cs vserver vserver-cs-exchange-https vserver-lb-exchange-oab -policyName pol-exchange-oab
bind cs vserver vserver-cs-exchange-https vserver-lb-exchange-rpc -policyName pol-exchange-rpc
bind cs vserver vserver-cs-exchange-https vserver-lb-exchange-eas -policyName pol-exchange-eas
bind cs vserver vserver-cs-exchange-https vserver-lb-exchange-aut -policyName pol-exchange-aut

Citrix and Microsoft guides, what's up with that?

Citrix recently announced an Implementation Guide for Microsoft Office 365 for Citrix XenApp and XenDesktop 7.x. After reading their recent deployment guides I noticed several mistakes which makes very clear to me that the writers may understand Citrix very well but not the Microsoft technologies and products. Now the purpose of these documents is to make sure that our customers and partners understand the technologies well enough to guarantee a successful implementation. I really wonder why they don't have a Subject Matter Expert proofreading the documents before they get released. Let me give a few examples.

The latest document starts with an error in the first paragraph:

Office 365 ProPlus includes a combination of online-based applications (Outlook, Word, Excel, PowerPoint and OneNote) that are accessed from anywhere via a web browser, as well as the latest traditional, locally installed version of Microsoft Office.

No it's not, Office ProPlus is the locally installed version of Microsoft Office.

Included with Office 365 ProPlus is an online email account with 50GB of storage and 1TB of file storage per user with OneDrive for Business.

No it's not, Office 365 ProPlus is not the name of one of the Office 365 subscriptions, it's the version of Microsoft Office you install on your computer.

Another example is Deploying Citrix NetScaler with Microsoft Exchange 2013 for GSLB.

image

Per service health checks, single unbound namespace, stretched DAG, nice! But why is there a CAS server in the DAG? Or more important, why is the DAG load balanced? When talking about Exchange high availability it's key to have a good understanding of the difference between Client Access HA and Mailbox data HA. The traffic to the Client Access servers is load balanced. Mailbox servers in a DAG enable the organization to deploy multiple copies of the database and store them on another DAG member server. The Client Access servers proxy the user's request to a Mailbox server.

Different clients add different suffixes to the domain name when they connect to virtual server. For example, /owa for web browser clients, /oa for Outlook anywhere.

That's the /rpc virtual directory, not /oa.

NetScaler GSLB also enables to maintain availability in case of site level disaster in which one of the sites is completely unavailable. This is shown in Figure 4. When there is hot sync between the mailbox servers and the user information is available on all the mailbox servers, then all the requests of site 1 can be completely served from site 2.

What is hot sync between the mailbox servers? I suppose the writer means that the database has a replica or copy on a server in the second site.

And then there is the Guide to Deploying Microsoft Exchange 2013 with Citrix NetScaler.

image

A snippet from a bit complex drawing. All servers, including the domain controllers are multi-homed. This is at least very uncommon and certainly not according best practices. And again there's a (single?) CAS server in the DAG that does not belong there. It should've been at least two CAS servers to demonstrate load balancing of the CAS servers.

And then there's the split-role/multi-role debate, where Microsoft recommends to deploy multi-role servers since the 2010 timeframe and still does for 2013. I guess the writers tried to update the document by writing "Exchange 2013 Multirole Servers MB/CAS" under the picture but forgot to actually update the diagram.

image

Is not going to work, oa needs to be rpc.

In the configuration shown above, a single namespace is used for all Exchange protocols. For example, for web access, the namespace is https://mail.ctxns.net/owa, and for Outlook clients the namespace is https://mail.ctxns.net/oa.

It's still /rpc, not /oa. And where is /Autodiscover, did we forgot this one?

More important flaw of the document is that the first half is the copy and past of some TechNet articles and some slides Microsoft presented at a tech conference, the seconds halve is just a series of screenshots. For me as an Exchange admin it does not really explain an good overview of load balancing with NetScaler, it does not dive in to the details as for instance session time-out settings, why we use least connection and if we can configure a slow start time to prevent an overloaded server after a reboot, transparent or non-transparent load balancing, routing considerations, etc.

And from a load balancer perspective I'm not sure how well the NetScaler guy understands load balancing an Exchange 2013 solution after reading the deployment guide. Especially with the small mistakes and bad representation of the architecture (mainly in the GSLB document).

Let's wrap-up my complaints with a positive remark. The latest version of the document is an improvement over an older version which had a dedicated set of Client Access servers in the perimeter network. I'm being sarcastic of course, I think there's room left for improvement. I mean the NetScaler is an interesting product, especially for customers looking for a CAG replacement. But when it comes to documentation and ease of configuration other vendors as KEMP Technologies have a better story. Personally I prefer an affordable load balancer with excellent documentation by a company who shows they understand their work loads over a powerful network appliance which is too complex to configure correctly and lacks proper documentation. This is where Citrix can learn from KEMP technologies.

Friday, March 20, 2015

Health Manager service not starting on Exchange 2013 Edge Transport servers

This is one of those issues in that was in Exchange 2013 RTM is is still there in CU8. I'm pretty sure it's very easy to fix and Microsoft PSS is aware of the issue, but anyway this is the error I'm talking about:

image

The Microsoft Exchange Health Manager service depends on the following service: MSExchangeADTopology. This service might not be installed.

This of course makes perfect sense because the Microsoft Exchange Active Directory Topology Service is not installed on an Edge Transport server. So in order to remove the dependency on the MSExchangeADTopology service we need to fire up Regedit and navigate to the following key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeHM. Open the Value DependOnService and remove MSExchangeADTopology from the Value Data.

image

Make sure the only remaining entry is eventlog and click OK to save. Reboot your server and verify that the Microsoft Exchange Health Manager is running.

Tuesday, March 17, 2015

Need to disable SSL3.0 for Exchange 2013 SMTP? Install CU8 to make it work

With the recent security issues with SSL 3.0 many organizations are in the process of disabling this vulnerable protocol for Exchange 2013 servers. Unfortunately SMTP appeared to keep using SSL 3.0 even after an administrator configured the server to use the more secure TLS 1.1 or 1.2 protocol.

This issue has been fixed in Exchange 2013 CU8. More information (a little bit) in the following KB article: SMTP is not transported over TLS 1.1 or TLS 1.2 protocol in an Exchange Server 2013 environment