Error 0xE0000100 when installing Server 2016 in a virtual machine

Today I ran into an issue while creating a new VM on a Windows Server 2016 Hyper-V host. The VM will run Server 2016 as well but throws an error message when Windows Setup loads:

Windows installation encountered an unexpected error. Verify that the installation sources are accessible, and restart the installation.
Error code: 0xE0000100"

I downloaded a fresh copy of the ISO, deleted and recreated the VM, rebooted but was not able to get rid of this error message. While researching I stumbled on this article: Windows Server 2012 R2 setup may fail on a virtual machine that is configured to use the minimum required memory

This article explains that this error message indicates insufficient memory. Although I assigned 1024 MB and not 512 MB I decided to add some more memory to the VM configuration.

To no avail, the same error occurred when I retried. Knowing now that this error indicates a low memory condition I decided to set the amount of RAM back to 1024 and disabled Dynamic Memory.  Started the VM and to my surprise, Windows Setup started and allowed me to install Windows Server 2016 on this VM.

At this moment I’m not sure why this happened. The Hyper-V documentation for Server 2012 R2 states that in this situation, a cold boot without an OS installed, the VM should have the Startup RAM amount available. This was initially 1024 and later 1500 MB, more than the 512 MB that Windows Setup requires.

When installing or upgrading the operating system of a virtual machine, the amount of memory that is available to the virtual machine during the installation and upgrade process is the value specified as Startup RAM. Even if Dynamic Memory has been configured for the virtual machine, the virtual machine only uses the amount of memory as configured in the Startup RAM setting. Ensure the Startup-RAM value meets the minimum memory requirements of the operating system during the install or upgrade procedure.

However, in my Dynamic Memory enabled VM I had little bit less than 512 MB available.

Hint: Pressing shift-F10 during Windows Setup opens a command prompt, allowing you access to tools such as diskpart, chkdsk and copy. Or in this case, allowed me to query WMI.

Now in this specific situation all memory of the VM host was assigned to running VMs. Because a restarting VM requires more than the minimum configured amount of memory during boot the Hyper-V hosts uses a disk-based paging feature called Smart Paging. However, Smart Paging does not work in a cold boot situation where the VM was powered off before booting the VM.

So when these requirements are met:

• VM with Dynamic Memory enabled
• All host memory assigned to running VMs
• Cold VM boot to install an operating system

The Hyper-V host is not able to make the Startup RAM amount of memory available and the VM sees the Minimum RAM amount. In this case this was 512 MB which triggered the low memory condition described in the KB article I mentioned earlier.

After freeing up some resources from this host the VM was booted again, now it showed the expected 1024 MB of memory available.

Server 2012 R2 or 2016?

Note that the KB article applies to Server 2012 R2 and I used the Server 2012 R2 Hyper-V documentation to learn more about memory management. From my reading there is no difference in behavior between Server 2012 R2 and Server 2016.

For reference, read:

• Windows Server 2012 R2 setup may fail on a virtual machine that is configured to use the minimum required memory
• Hyper-V documentation for Server 2012 R2

Friendly reminder: /RecoverServer to an older OS version is not supported

In case you missed it, it’s currently not recommended to deploy Exchange 2016 on Windows Server 2016. The Windows Server team added a section dedicated to Exchange in their Windows Server 2016 Release Notes:

If you attempt to run Microsoft Exchange 2016 CU3 on Windows Server 2016, you will experience errors in the IIS host process W3WP.exe. There is no workaround at this time. You should postpone deployment of Exchange 2016 CU3 on Windows Server 2016 until a supported fix is available.

It’s expected that a blog post with a similar message will appear soon on the Exchange Team Blog. The difference with the first post is that this post has to include recommendations to customers that deployed Exchange 2016 on Windows Server 2016, which is supported since CU3. As the Windows Server team already stated, there is no fix or workaround available today.

To make matters worse, it looks like this issue only applies to Server 2016 servers that have been added to a DAG. Good news for customers with stand-alone Exchange 2016 on Windows Server 2016 servers but it makes recovery a tad bit more complicated.

The supported approach would be something like this:

1. Stand up new Windows Server 2012 R2 servers
2. Install Exchange 2016 CU3 (or CU2 to prevent this from happening)
3. Optional: Add both servers to a DAG (not the same one, 2012 R2 and 2016 servers can’t be members of the same DAG)
4. Move the mailboxes to the new servers
5. Uninstall Exchange 2016 from the impacted servers

The big question here is, will the admin be able to perform all these tasks with Exchange 2016 being in this state: Exchange Server 2016 & Windows Server 2016 Datacenter IIS AppPool's constantly crashing. ECP/Powershell inaccessible

A user in a community recommended this approach:

There is no fix at the moment. Therefore if you have put Exchange 2016 on to Windows 2016 then the only option is to shutdown the server, rebuild as Windows 2012 R2 then do a recover server install of Exchange 2016. That works and allows you to remove it cleanly.

Unfortunately this process is not supported (*) and never has been. This can be found in Recover an Exchange Server:

What do you need to know before you begin?
The server on which recovery is being performed must be running the same operating system as the lost server. For example, you can't recover a server that was running Exchange 2013 and Windows Server 2008 R2 on a server running Windows Server 2012, or vice versa. Likewise, you can’t recover a server that was running Exchange 2013 and Windows Server 2012 on a server running Windows Server 2012 R2, or vice versa.

So deploying a new server with Server 2012 R2 and then install Exchange 2016 with the /RecoverServer option is not supported.

Let’s see what the Exchange team will have to say on this.

(*) Not supported in this context means that Microsoft Support cannot support your environment if you do this. The exception is if Microsoft Support asks you to follow this process, in that case you will be supported.

Microsoft Teams, here’s the admin guidance

Yesterday Microsoft released a preview version of Microsoft Teams, the new product to compete with Slack. It will be very interesting to see Teams integrate in Office 365 as yet another collaboration product next to Skype for Business, Exchange, Yammer, SharePoint Newsfeed and Office 365 Groups.

Currently Microsoft Teams is not enabled by default, a tenant admin has to enable the application in the Office 365 Admin Center.

This will change in 2017 Q1 when Teams will be enabled by default. Or as Microsoft states in the Message Center (id MC84541):

We are rolling this preview out off-by-default, to give you a chance to explore the new capabilities. This experience will be turned on-by-default in the first quarter of 2017. At this time, you can control access to this experience, at the organization level. We are working on user-level controls for this feature, and will communicate again when available.

I know that many of my customers are still struggling how to handle Office 365 Groups creation and the lack of governance and control around this area. Microsoft is working hard to improve this, as they explained in this session at Ignite 2016: Manage Microsoft Office 365 Groups But with every new Microsoft Team there will be a corresponding Office 365 Group created.

This raises the question how admins can manage the Microsoft Teams roll-out in their organization. What are the firewall requirements? And how to estimate bandwidth for peer-to-peer calling?

The good news is, there is actual admin documentation. The bad news is that it’s no longer consolidated on a single place as we’re used to with TechNet. The currently available resources for admins can be found here:

• Microsoft Teams - Microsoft Tech Community
• Frequently asked questions about Microsoft Teams - Office Support
• Microsoft Teams: Step-by-step intro for using, enabling and managing the experience - Microsoft Mechanics YouTube Channel
• Introducing Microsoft Teams (in Preview) - Microsoft Virtual Academy

Much to my surprise the most relevant information for admins was hidden in part 2 of the MVA video series. I highly recommend to download the Deploy_Microsoft_Teams.pdf document that can be found in the Resources section of this training.

There’s a ton of extremely valuable information in this document that helps you understand how to prepare for Microsoft Teams, at least from an infrastructure point-of-view.

Have fun!

Office 365 MFA is awesome! Unless you’re an administrator…

For some reason I have never worked with MFA with Office 365 until last year. And I must say, it is awesome! Even the free version of Azure MFA that’s included with the Office 365 subscription meets the requirements of most organizations. It’s very easy to setup and configure and the end-user experience is pretty good too, supporting text messages, phone call or the Azure Authenticator app.

Microsoft did a great job integrating the PhoneFactor acquisition (2012) in Azure AD and Office 365. So it’s not a surprise that a lot of organizations plan to enable MFA for all users, some users or the users with an admin role. And that’s where the issue is, Office 365 MFA currently does not support Remote PowerShell. Or I should say Remote PowerShell does not offer support for MFA because this would require support for Modern Authentication. This applies not only to Exchange management, but too PowerShell management of SharePoint, Skype for Business, EOP and Security & Compliance as well.

How about app passwords then? We can use app passwords for applications that do not support MFA right? Unfortunately app passwords are not working either.

When talking with Microsoft Premier Support they explained there’s currently no news to share. However, a Microsoft Most Valuable Professional explored the limits of his NDA on Facebook when he disclosed that Microsoft has made a preview version of Exchange PowerShell to beta testers at the moment. I’m very keen to learn more about this new version, because currently Remote PowerShell depends on the version of PowerShell that’s installed in the OS of the workstation. I’m assuming that MFA support requires the installation of additional software.

Ironically the new Office 365 Secure Score site (https://securescore.office.com/), a challenge were organizations receive points for increasing the security, awards 50 points for organizations that enable MFA for all their Tenant Admins. There’s no mention that this removes the ability to manage Office 365 with PowerShell.

Keep an eye on the Office 365 Roadmap and the Azure MFA Documentation for updates.

Microsoft reverses the undocumented changed Exchange Online license removal behavior

A couple of weeks ago some users reported a change in behavior when an Exchange Online license was removed from an Azure AD user object. It was reported that with the changed behavior mail would still flow to a mailbox, after the license was removed. When a customer opened a service request with Microsoft Support, they acknowledged the change but there was no documentation available, nor was this change announced on the Office 365 Roadmap.

The first public information appeared early October when Microsoft added the following text in the License Removal section of the Delete or restore user mailboxes in Exchange Online document.

Previously, in Exchange Online, if you removed an EXO license the user was kept, but the mailbox user was transformed into a mail user and the mailbox was moved to the recycle bin. You could then recover the mailbox within the 30 days time limit.

This has now changed in Exchange Online. If you remove a user's license, the user mailbox will no longer be able to sign in and use Exchange Online or Office 365. The user mailbox will remain in Exchange Online until it is deleted, permanently removed or purged by the Office 365 admin. You can reassign a license to the user and make the mailbox active again.

But if you look in this document today the text was removed and replaced with a link to this post on the Official Exchange Team Blog: Change in behavior for delicensed Exchange Online users

In this post Microsoft explains the change in more detail, apologizes for the way they handled the change and most important, that they rolled back the change for now:

Due to extensive feedback from customers we are rolling back this feature to the original behavior in order to improve the feature before we release it again in an easier format (and with better documentation).

For more information, read the full post here: Change in behavior for delicensed Exchange Online users

Microsoft and their frequent product name changes

Today I learned that the Microsoft Federation Gateway was rebranded to Azure Authentication System some time ago. This reminded me of that one time when we made a marketing video and had the narrator use the name Windows Azure when Microsoft decided to rebrand the service to Microsoft Azure. We had to pay the voice actor again to record the text for a second time, now with Microsoft Azure.

Sometimes a new name makes perfect sense, for instance when Microsoft renamed the RPC over HTTP protocol to Outlook Anywhere. Or when they rebranded Business Productivity Online Suite (deskless worker anyone?) to Office 365.

In the Exchange space the favorite buzzword today seems to be Modern. The newly architected Public Folders in Exchange 2013 and Exchange Online became Modern Public Folders. The smoother ADAL based authentication became Modern Authentication and I overheard someone jokingly using the term Modern Outlook Anywhere for MAPI/HTTP.

Smart branding can add tremendous value to a product, but inconsistent and frequent renaming will confuse customers and hurt the recognizability of a product. An example that comes to mind is the product we know today as Skype for Business, this product received a new name with every single release. Another bad example is the recent rebranding of the Exchange web interface to Outlook on the web. I always grin when I find another ‘Outlook Web App’ in an Exchange 2016 TechNet article. It looks like even the technical writers have a hard time keeping up with the name changes. I can’t blame them.

Focused Inbox for Outlook is delayed

When Microsoft announced Focused Inbox for Outlook and Outlook on the Web (OWA) in July they planned to release the new features to First Release customers starting early September 2016. Roll-out to the 4th ring of customers was scheduled for October.

One month between First Release and GA may seem much, but for organizations that need some more time to understand the impact and communicate the changes with the end-users, a month isn’t that much time.

But more importantly, September has already passed and we have not seen the new feature to appear nor any new updates on the Office Blog or documentation for administrators.

Last week I attended Microsoft Ignite in Atlanta and had the pleasure to speak with some members of the Outlook time. My understanding is that the feature is ready, the documentation has been written but the actual deployment has been rescheduled to the November/December timeframe.

Microsoft has promised to do a better job than they did with the roll-out of Clutter. Documentation for admins should be published before launch to First Release tenants. If you want to be prepared, read up on Focused Inbox admin controls in my previous article.

On-premise or on-premises?

It’s on-premises.

Exchange Online and the new email address limit

Update: The actual limit was updated to 400, as documented in “Too many proxy addresses" error when you try to add or remove email addresses from an Exchange Online recipient.

Exchange Online, just as any other cloud service, is a shared environment where resources are pooled between multiple tenants. This means that certain limits need to be enforced, either to ensure that the services is being used as intended or to prevent that some users consume an uneven share of the available resources.

Luckily most limits, not all, are documented quite well in the Exchange Online Limits section of the Office 365 Service Descriptions. One of the tables on that page contains some limits with regards to recipients. See the following screenshot of this table as it was on the 10th of July, 2016: (click to enlarge)

And here is what recently changed:

A new Recipient proxy address limit was added to the table and immediately enforced.

Interesting is that the column for Exchange 2013 is populated with the value of 200 now too:

Unless a recent CU introduced a hardcoded limit I don’t think this is accurate. By my knowledge the real limit in the on-premises world is the character limit of the proxyAddresses AD attribute.

Now this may not apply to you, but there are an awful lot of people out there who have up to 300 or more proxy addresses. Some users created custom addresses for each mailing list of vendor account as Exchange never implemented a wildcard email address feature (jetzemellema+amazon@gmail.com).

And to make matters worse, the admin interfaces do not allow to remove individual email addresses and then save the object again. A possible work around is to export all proxy addresses to CSV, remove them all, clean up the CSV to contain <200 entries and add them again with PowerShell.

The easiest long-term solution appears to be to add additional Distribution Groups where your mailbox is the only member. Now add a bunch of those addresses to the DG to ensure you can still receive all messages sent to the addresses.

In hindsight this would’ve been a perfect topic for Microsoft to announce before implementing the change, including guidance for customers who are impacted by this change.

Make your HCW experience even more fun

Ever wondered what happens when you click through the new and shiny Hybrid Configuration Wizard? Wouldn’t it be awesome to be able to see what happens when you wait in real time? Now you can.

People with a background in Unix or Linux are probably familiar with the tail program. tail reads the output of a file and keeps doing so when the file is updated with new data. This is an ideal tool to view log files in real time.

PowerShell offers similar functionality in Get-Content with the -Wait switch. With that in mind, all we need to do is find the most recent log file in the directory as every instance of the HCW creates a new log file and then read the contents of that file.

Start the HCW first, we need the log file to be there before we can read it, and then enter the following one-liner in PowerShell:

Get-Item "$ENV:appdata\Microsoft\Exchange Hybrid Configuration\*.log" | Sort LastWriteTime | Select-Object -Last 1 | Get-Content -Wait

You like that? Then try using Get-Content -Wait against C:\ExchangeSetupLogs\ExchangeSetup.log the next time you’re installing or upgrading Exchange. Have fun!

How to fix ALT+S in Firefox

Key combination ALT and S is commonly used to save or submit data in web application, such as Exact Online, phpBB and vBulletin. Since Mozilla Firefox 2.0 this is not working anymore.

This can be fixed by editing two settings in the advanced settings of Firefox.

• In Firefox, visit about:config
• Change ui.key.chromeAccess to 5
• Change ui.key.contentAccess to 4

The changes are immediately effective, no need to close and reopen the application.

Update, fixed: KB3176934 breaks remote PowerShell

Update: This issue has been fixed in the re-released KB3176938 update.

Today I ran into an error message on one of my systems. PowerShell was unable to import my remote session to Exchange Online.

Import-PSSession : Could not load type 'System.Management.Automation.SecuritySupport' from assembly 'System.Management.Automation, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'.

A quick Google search learned that this KB3176934 update was released a couple of days ago and is known to break DSC, remote PS and probably other stuff.

Microsoft scheduled an updated update to be released on the 30th of August 2016. If you can’t wait for some reason, for instance you planned to do some actual work today, uninstall the update and reboot your system.

wusa /uninstall /kb:3176934

Source: PowerShell DSC and implicit remoting broken in KB3176934

Focused Inbox admin controls appear in Exchange Online

Back in 2014 Microsoft acquired Acompli, a company that had developed the popular mobile apps with a feature called Focused Inbox. A server side algorithm was used to “learn” the difference between important email and less important email, providing the users a very clean view of their mailbox showing only the most relevant messages.

The Acompli apps have then been rebranded to Outlook Mobile and the algorithm was migrated to Office 365 and Azure’s machine learning capabilities. The next step is to bring Focused Inbox to Outlook and Outlook on the Web, which Microsoft recently announced. See Outlook helps you focus on what matters to you.

I’m sure that any Exchange Online admin remembers how Clutter was introduced, a new and potentially confusing mailbox feature without any admin controls. With Focused Inbox Microsoft is planning to do a better job and has announced admin control before the actual roll-out to the Office 365 tenants.

Admins will be able to disable or enable Focused Inbox on the tenant level with Set-OrganizationConfig and the -FocusedInboxOn parameter. Similar to Clutter there will be cmdlets to manage the feature per mailbox as well, expect something like Get-FocusedInbox and Set-FocusedInbox.

Focused Inbox will begin to roll-out in the September-October timeframe, starting with First Release customers. More information on admin controls will be available before roll-out, giving admins more time to develop a strategy on how to handle the implementation of this new feature.

Outlook 2013 June 2016 update causes Mail applet to stop working

Many issues with opening the Mail applet in the Control Panel have been reported in the technical communities recently. Apparently this applies to Office 2013 Click-to-run (C2R) installs with the most recent ‘June 2016’ update installed. The build number of the affected installs is 15.0.4833.1001 and newer.

Microsoft is aware of the issue and will have this fixed in the upcoming July 2016 update. While the issue prevents the user from opening the Mail applet in Control Panel, there are several workarounds to access the Outlook profile settings to either select a different or create a new profile, or to open the Control Panel applet to edit existing profile settings.

Method 1

Start Outlook with the /profiles switch:

Outlook.exe /profiles

Method 2

Toggle the ‘Prompt for a profile to be used’ setting with a registry key.

HKEY_CURRENT_USER\Software\Microsoft\Exchange\Client\Options\PickLogonProfile

Value type = REG_SZ
1 = "Prompt for a profile to be used"
0 = "Always use this profile"

Method 3

Add two missing registry keys:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Control Panel\Cpls]
"MLCFG32.CPL"="C:\\Program Files\\Microsoft Office 15\\root\\office15\\MLCFG32.CPL"

[HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\Windows\CurrentVersion\Control Panel\Cpls]
"MLCFG32.CPL"="C:\\Program Files\\Microsoft Office 15\\root\\office15\\MLCFG32.CPL"

Method 4

Revert the Office 2013 C2R install to 15.0.4823.1004, the May 2016 update.

1. Exit all Office applications.
2. Open an elevated command prompt. To do this, click Start, type cmd in the Start Search box, right-click Command Prompt or cmd.exe, and then click Run as administrator.
3. At the command prompt, type the following command, depending on your bitness of Windows, and then press Enter:
   For an Office 2013 installation and a 32-bit version of Windows:

   cd %programfiles%\Microsoft Office 15\ClientX86

   For an Office 2013 installation and a 64-bit version of Windows:

   cd %programfiles%\Microsoft Office 15\ClientX64

4. Type the following command, and then press Enter:

   officec2rclient.exe /update user updatetoversion=15.0.4823.1004

5. When the repair dialog box appears, click Online Repair.
6. Click Repair, and then click Repair again.
7. After the repair is complete, start Outlook.
8. Click File, and then click Office Account.
9. In the Product Information column, click Update Options, and then click Disable Updates.
   Important note This step is very important. The repair process re-enables automatic updates. To prevent the newest version of Office Click-to-Run from being automatically reinstalled, make sure that you follow this step.
10. Set a reminder in your calendar for a future date to check this Knowledge Base article (3175861) for a resolution for this issue. Enable automatic updates in Office again after this issue is fixed. Enabling automatic updates again will make sure that you don't miss future updates.

KB article in the making

The instructions in Method 4 mention a KB article with id 3175861. Unfortunately that article was announced but has not been published yet.

To learn more about this issue I recommend to check https://support.microsoft.com/en-us/kb/3175861 in a few days to read more.

Multiple Transport Rule conditions and the OR operator

Here’s something I ran into today and would like to share. Exchange transport rules, also known as mail flow rules, can have multiple conditions, actions and/or exceptions which makes them flexible and a powerful tool. However, if you add multiple conditions an AND operator will be applied. This means that the rule will be triggered only when all conditions are True.

How can we replace the AND with an OR? For instance, if we want to apply a certain action when the sender is member of a group or a specific person? The answer is that we can’t do this with a single transport rule. There is an easy solution, simply create a copy of the transport rule and update the condition. Now the action will be applied when either of the transport rules is triggered because the single condition is True.

Exchange 2016 courses on MVA, edX and their quality

Yesterday Tony Redmond published an article titled Virtual academies, odd questions, and MCSE recertification. In the post he shows numerous examples of bad worded questions and incorrect or outdated answers on Microsoft’s MVA platform. The Exchange 2013 and Exchange Online content on MVA could definitely use a thorough upgrade.

On May the 3rd the Exchange Team announced new Exchange 2016 material: Exchange Server 2016 Online Training Courses Now Available! Most notable was that the four courses were presented of the edX platform instead of their own MVA, not at least because the edX courses have cost $ 49 each.

Today I walked through the first course: Microsoft Exchange Server 2016 - 1: Infrastructure, which is free as long as you don’t require a certificate, to get an idea of the quality. My first impression is that the quality is not the worst I’ve ever seen, but there is a lot to improve. First let’s take a look at the first two modules and check for factual errors. Make sure to continue reading because there is more…

Module 1: Exchange Server 2016 Prerequisites and Requirements

This information seems to be taken from the Exchange 2007 documentation: Planning Processor Configurations. Both the 1.000 mailboxes per CPU core as well as the Average profile of 10 messages sent and 40 received are from the Exchange 2007 timeframe.

 

The Exchange 2016 sizing guidance refers to the article for Exchange 2013. There we can read that the per mailbox memory requirements for the 50 and 100 messages profile are 12 and 24 MB, not 3 and 6 MB as stated in the course.

 

This command is going to fail because of the dot after -Restart.

 

By al means, do not install any version of WMF later than 4.0. Recently WMF 5.0 was released but this new version is currently not supported with any version of Exchange. An no, the asterisk does not refer to anything.

 

This command is going to fail because of the space after RSAT.

 

Now this is an interesting question, the answer is ‘hidden’ in the title of the question.

Module 2: Exchange Server 2016 Deployment

The UM role was integrated with the Mailbox server role beginning with Exchange 2013, not 2016.

 

Single-server recommended to run in a VM? I fully agree, but never heard this recommendation form the Exchange team. And replicate the VM to another Hyper-V server? Hyper-V Replica is NOT supported for Exchange.

 

It’s not, by default there’s a V15 folder in that path under where Exchange is installed.

 

This command will fail because the /mode switch is missing.

 

The correct answer is EdgeTransport, no space between the words.

 

The correct name was Forefront Online Protection for Exchange (FOPE). I said was, because FOPE was replaced with Exchange Online Protection (EOP) a couple of years ago. Forefront Online Protection was never the name of a product or service.

Due to time constraints I decided to stop after the first two modules.

But wait, they are on MVA too!

Initially I wanted to explain how odd it is that Microsoft used the edX platform instead of their own MVA. But when researching for this article today I discovered that the exact same courses have been published on MVA just yesterday. And when I say ‘exact same courses’, I mean the same content but now presented in a video of two people reading the same course.

Different format, same content and same errors (WMF 4.0 or later):

For me personally this format of video learning does not work at all, because the pace is too slow. I prefer to read on my own pace and be able to skip some content when I’m already familiar with a topic. But if the video format works for you, use the MVA ones and save $ 49 per course.

In conclusion

The majority of the content in the first two modules of the first course was copy and pasted from the TechNet Library and did not add any value for experienced Exchange administrators. Paid courses in a better format are on edX, the free version is on MVA as a video. Pick one that works for you.

Be aware that the learning content contains errors and more authoritative information on the topics can be found in the TechNet Library as well on the Exchange Team Blog. As the guidance and features change with every CU or Exchange Team blog post, expect the quality of the learning content to get worse over time.

The new HCW on Exchange 2010, a few notes

Today I used the new Exchange 2010 Hybrid Configuration Notes in a production environment and wanted to share my notes. This is not an extensive review of the new HCW, just a few short remarks.

First of all, Exchange 2010 Update Rollup 13 replaces the button to open the old HCW in EMC with a link to the download page for the new HCW. If you’re not ready for the new HCW and want to do additional testing, do not upgrade the CAS server where you’d execute the HCW yet to UR13.

The new HCW requires .Net Framework 4.5 which is typically not installed on an Exchange 2010 server because Exchange 2010 uses version 3.5. Make sure the latest updates are installed after installing 4.5 on the server.

My contacts at Microsoft assured me that the new HCW would operate just as the old one did, but better. Testing discovered that this is not entirely true. The new HCW creates Send Connectors and Organization Relationships with different names than the old HCW did. If pre HCW and post HCW scripts are being used to correct the shortcomings of the HCW they need to be updated to use the new names that now contain a GUID. Common tasks after running the HCW are changing the -TargetOwaURL parameter of the Organization Relationship or update the Send Connector to use one or more Edge Subscriptions instead of an HT server.

The page to edit the Hybrid Domains has improved a lot. Unfortunately it’s not possible to sort on enabled status or domain name by clicking on the column header. This makes locating a domain very hard, especially when you’re managing a couple of hundred accepted domains.

The new log file is much more verbose, but you won’t find it in the most logical places. The new location is $ENV:appdata\Microsoft\Exchange Hybrid Configuration. Tip: search for the string *ERROR* or WARNING. That’s correct, the string ERROR is enclosed by double quotes, WARNING is not.

Leaving feedback is much easier with the Give feedback link on every page of the wizard. Unfortunately the HCW freezes for some minutes after sending the feedback, but be patient and the HCW can be continued.

Office Online Server released, confusion around sizing

In case you missed it, the Office team is in the process of releasing the RTM version of Office Online Server (OOS) to the public. Customers with a Volume Licensing account can download OOS from the Volume License Servicing Center, OOS will be available on MSDN beginning May 9th, 2016.

For most Exchange admins OOS as well as the previous versions of the same product, are a new technology. For a great overview of deploying Exchange 2016 with OOS I recommend to view the recording or at least the slides of the session that Michel de Rooij recently presented on this subject.

Unfortunately the documentation for OOS is not (yet) of the high standard we’re seeing with Exchange and some other products. In this post I want to highlight two topics as an example: sizing requirements and virtualization support.

Sizing your OOS servers

Maybe the comparison with Exchange is not the best example here, because Exchange 2010 was the last version where sizing documentation was of a very high quality. For recent versions of Exchange the guidance is shifting towards using the calculator to design your environment, instead of using the calculator to validate your design.

The guidance for OOS is even worse:

That’s odd, SharePoint 2016 is a very different application and the recommended production architecture is to spread the roles over multiple servers. SharePoint does know the Single-Server farm concept but this is recommended for development, testing or very limited production use. The SharePoint teams gives two sets of minimum requirements, one for development and one for pilot or user acceptance scenario’s:

We’re sizing our production OOS deployment so let’s pick the largest one: 4 CPU cores and 24 GB of memory. The assumption here is that the Office team had the SharePoint Single-Server deployment in mind when they referred to SharePoint sizing for OOS.

But wait, there is another authoritative source: the Exchange team! In the Exchange 2016 Preferred Architecture is a short section dedicated to designing your OOS servers.

So without asking any questions about the number of users, % of OotW usage or whether we need view-only or editing capabilities we’re now at 8 CPU cores and 32 GB of memory, times two per datacenter of course because the PA assumes HA. Please note that the SharePoint team recommends to use at least double of your memory as the free disk space, so that would make 64 GB instead of 40.

With the current lack of real-world performance figures it probably would make sense to start with a relatively small server, monitor your deployment carefully and add resources if necessary. Which brings me to my next point.

Virtualization

Just as every other modern application OOS supports deployment in a virtualized environment, giving customers the choice and flexibility to deploy OOS on their own terms.

The first bullet is probably good advice for performance and manageability reasons, the second bullet is basic common sense. The interesting part is hidden in the first paragraph:

…is supported when you deploy it using Windows Server Hyper-V technology…

Is Microsoft really saying that you’re allowed to deploy OOS on Hyper-V but not on VMware, Xen, KVM or any other hypervisor solution that is certified through the Windows Server Virtualization Validation Program (SVVP)? Yes they are, but this has to be a mistake. I cannot think of any valid reason behind this statement.

But wait, there is more…

While researching this subject I noticed several other interesting or questionable statements in the OOS documentation on TechNet. To name a few:

The Office team recommends SSL offloading, that means that the load balancer would be the endpoint for the SSL tunnel and that all traffic between the load balancer and the real servers will be unencrypted. This goes against the security principle of treating both external as well as internal networks as unsafe by default. It’s considered best practice to deploy SSL bridging instead. The Office team acknowledges this and recommends to mitigate the risks involved by recommending the use of firewalls and private subnets to secure the traffic.

The load balancing section mentions a requirement for layer 7 routing and client affinity but lacks any recommendations on what affinity options to choose and does not mention how to configure the load balancer’s health checks. In practice we see that a lack on guidance in this area generally leads to bad implementations.

In conclusion

I could go on for a while, but I won’t. I recommend every Exchange organization considering OOS with Exchange 2016 to perform a cost-benefit analysis to start with, for instance if 95% of the users will use non-OotW clients to access Exchange 2016 mailboxes an OOS deployment maybe doesn’t make sense. And there is of course the licensing aspect, as editing capabilities are not free and are coupled to Office suit licensing.

I you are planning your OOS deployment with Exchange 2016, make sure to contact your Microsoft representative to confirm that OOS on your hypervisor will be supported. From a sizing perspective, start with a small VM and add resources when necessary. And make sure to keep an eye on the Twitter an Blog-o-sphere for more updates on this subject.

Exchange 2016 admins, prepare for Office Online Server

Update may 10th, 2016: OOS now available on MSDN!

Support for in-line viewing and editing of attachements in Outlook on the Web was one of the (few) major updates when Exchange 2016 was released. Unfortunately the required Office Online Server (OOS), formerly known as Office Web Apps Server, has not been released yet.

This may change soon as Microsoft is starting to relaese the bits to the MSDN subscriber downloads portal. A categorie for Office Online Server was added, containing just an OOS Language Pack.

With the recent release of SharePoint 2016 RTM it is expected that OOS will be released anytime now. While we wait, let’s read up on OOS in the TechNet Library: Office Online Server.

What is the new Office 365 SPO address type?

Since a couple of days Office 365 customers are reporting that they notice a new SPO address type appearing at some of their user’s mailboxes.

The SPO initialism indicates the new address type is related to SharePoint Online features, the fact that it only appears on objects with a SharePoint Online license confirms this.

At this time there’s no public documentation that describes the function of this new addresses. If you happen to know more, please leave a comment.

Exchange Hybrid? Microsoft has no plans to make creating shared mailboxes easy.

In two earlier posts (one, two) I wrote about the limited options to provision shared mailboxes in a hybrid environment. Or more specific, in an environment with directory synchronization. In short, it’s not possible to create shared mailboxes or convert regular mailboxes to shared in Exchange Online.

While both New-RemoteMailbox and Set-RemoteMailbox support the -Type parameter,  but it will only accept Regular, Room or Equipment as values and not Shared. We asked Microsoft to reconsider and add support to create remote shared mailboxes. Unfortunately the Design Change Request (DCR) was rejected. No specific reason was given but indicated was that our request was the first and only ask for this feature.

Earlier, when we suggested to remove the Convert to shared button from EAC Microsoft stated they considered customers wanting to convert a mailbox to shared a ‘niche scenario’. If you disagree and think customers should be able to provision and convert to shared mailboxes, make sure to let Microsoft know. Managed customers should ask their TAM, for smaller customers I’m afraid they need to burn a $499 support call as I’m not aware of another channel to add your request to Microsoft’s database.

For now this means that new shared mailboxes need to be provisioned on-premises and then be moved to Exchange Online. To convert a mailbox in Exchange Online we need to move it back to on-premises, convert the mailbox and then move it to Exchange Online again. Or read my work-around: Convert a user mailbox to shared in a hybrid environment.

PowerShell 5.0 re-released. Do not install on Exchange!

Two weeks ago Microsoft decided to offer the latest version of .Net Framework as a recommended update. Many Exchange admins found out the hard way that it’s not wise to install every single update without checking if it is actually supported to run them in combination with Exchange. In case of .Net Framework 4.6.1 there were in fact known issues, as some people soon discovered.

Today Microsoft re-released Windows Management Framework (WMF) 5.0 RTM. WMF included PowerShell 5.0 which brings many new features. Advanced administrators are probably looking forward to install WMF 5.0 on all their systems as soon as possible. But don’t do that, not before you’re absolutely sure that it is supported with Exchange.

This information can be found in the Exchange Server Supportability Matrix, one of the most important Exchange resources that’s often overlooked. In the ESSM we find for instance that .Net Framework 4.6 is not supported:

And the same applies to WMF 5.0:

And for customers with Outlook 2007 who consider Exchange 2016:

By the way, it’s perfectly fine to use PowerShell (WMF) 5.0 to connect to Exchange Online. In fact, if you’re on the November update of Windows 10 (Version 1511) this means that PowerShell 5.0 is already installed on your system.