Exchange 2013 OWA & ECP failure after deploying CU3

Dear gents, this site is still under construction. It will updated soon.

Acronym:
CU                         Cumulative Update
UR/RP                   Update Rollup/Rollup Pack
Ex2K13                  Exchange 2013
KB                          Knowledge base
AD                          Active Directory
URL                       Uniform Resource Locator
HTTP                     HyperText Transfer Protocol
HA                          High Availability

CU3 [Cumulative Update 3 for Exchange Server 2013 (KB2892464)] was released in November 2013. In this article, the content of the update will not be described. A complete list of customer-reported issues resolved in Exchange Server 2013 CU3 can be found in Knowledge Base Article KB 2892464.

With any CU, RP or other update, there may sometimes be difficulties after installation. Maybe this sounds familiar to some administrators out there.

Some issues may occur:

  • Broken virtual directories
  • Powershell shortcuts missing
  • Powershell dll’s missing
  • References missing in registry
  • ASP.net not correctly updated or broken
  • OWA, ECP or ActiveSync not working
  • Certificate problems
  • Etc …

In the case of the Exchange 2013 CU3 and above, the following functionalities stopped working.

  • OWA
  • ECP

Deployment Guide CU3 and above

CU deployment is equal to a full exchange server installation. There is a good  blog, here, on how to install/deploy this update.

If you prefer scripting or don’t want to loose time typing, use the following script to put the servers in maintenance mode.

Symptoms

After deploying Exchange 2013 CU3 and above in DAG or non DAG mode, the users cannot access OWA or ECP although outlook-anywhere is working correctly.

pic2

pic1

Errors:

I’ve noticed in some environments an absence of error or log file entry (unless we couldn’t find it 🙂 ).

In some cases, you’ll find this kind of event ID.
Event ID: 4 & 1309
source: ASP.Net

Or in OWA/ECP = error 500

pic5

pic6

Note: normally you should also see an error message in OWA indicated by, …canary…,  .

We tried everything to solve this problem without success.

  • Updated only 1 node to CU3, the users on the PAM server couldn’t access OWA/ECP.
  • Activated the updated node, the users couldn’t still access OWA/ECP.
  • The 2nd node updated to CU3, with the  same problems
  • The DB activated on the 2nd node, with the  same problems
  • Created new DB’s on both DAG nodes, moved some users, with the same problems
  • Created new users’ and mailboxes on old and new DB’s, with the  same problems
  • Created new virtual directories, with the  same problems
  • Verified IIS settings, …
  • Installed new Exchange 2013 server with CU3, with the  same problems

Cause

Web site protection: URL rewriting, canary URLs to improve security

The main reason why it is failing is because of a “canary” token-mismatch between client and server.

The canary attributes are created during the first Exchange 2013 schema preparation.

Basically a canary is a secret token between the client and server in OWA, ECP or other web services which get stored in the cookie collection of your browser and then get submitted with the various requests that your browser sends. For any request, the value of the GUID stored in the URL is compared to the one stored in session state. If they don’t match, or if the GUID is missing from the URL, the request is considered malicious and it is blocked, the originating IP address is also logged. Eventually the user will get an http error message 500: Internal Server Error, and the server encounters an unexpected condition which prevents it from fulfilling the request.

AD schema: Exchange canary attributes

 pic7

  • ms-Exch-Canary-Data-0
  • ms-Exch-Canary-Data-1
  • ms-Exch-Canary-Data-2

To be frank, I’m not 100% sure due to which value(s), attribute(s) or service(s) the token verification is failing.

The only valuable attribute what I can see in RTM, CU2 and CU3 (still a guess) that can cause a canary failure, but cannot find detailed information, is the attribute below:

  • Global catalog attributes are added by Exchange 2013 RTM: ms-Exch-Sync-Cookie
  • Attributes modified in Exchange 2013 CU2: ms-Exch-Sync-Cookie
  • Attributes modified in Exchange 2013 CU3: ms-Exch-Coexistence-Secure-Mail-Certificate-Thumbprintms-Exch-Sync-Cookie

Is this attribute the cause of canary failure on OWA and ECP? We have to address to MS to be sure.

Scenarios

Note: the exactly point of failure has to be validated by MS. I guess the failure is on 6 or/and 7 due to modified attribute above.

Working scenario:

pic3

  1. Get OWA
  2. Username and password
  3. If login successfully
  4. Session ID will be created by the server
  5. A cookie will send to the user with session id
  6. Server creates shared secret token (canary)
  7. The canary & session ID are saved in session store state
  8. The canary is rewritten into URL and send to the user
  9. Browser will open the requested page

Not working scenario:

pic4

  1. Get OWA
  2. Username and password
  3. If login successfully
  4. Session ID will be created by the server
  5. A cookie will send to the user with session id
  6. Server cannot create shared secret token (canary)
  7. (or) The canary & session ID aren’t saved in session store state
  8. The canary is not rewritten into URL and send to the user
  9. Browser will return error message 500

Resolution

Note: AD backup is required before taking actions

Applies to:  Exchange 2013 all CU’s

Featured image

  1. Open ADSIEDIT.msc
  2. go to CN=Client Access,CN=”your organization name”,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=”domain”
  3. right click “CN=your Client Access Server” or “CN=your single -or multi-role server(s)” and open properties
  4. scroll down until you find msExchCanaryData0, msExchCanaryData1, msExchCanaryData2 and don’t forget the subfolders, there are also canary values stored
  5. Copy all of these values to a text file with their attribute names and paths.
  6. Once copied and saved, erase all these values
  7. Logon on the CAS server(s) or multi-role server
  8. Open IIS management
  9. Go to the application pools
  10. Right click on “MSExchangeOWAAppPool” and click on recycle
  11. Note: during a recycle the established sessions will be lost, but in this case no worries.
  12. Cmd: appcmd recycle apppool /apppool.name:”MSExchangeOWAAppPool”
  13. If single role exchange architecture, reboot all mailbox servers with CU3 deployed
  14. If multirole, just reboot the server and skip recycling.

Cheers,

Kyn

Advertisements

10 thoughts on “Exchange 2013 OWA & ECP failure after deploying CU3

  1. Hi,

    Many thanks for this beautiful article.
    I just want to clarify one point. This issue is not due to CU3 only, because we can see in the following topic that the problem occurs too in older version of Exchange 2013 : http://social.technet.microsoft.com/Forums/sharepoint/en-US/777b51ee-330d-43cc-a56e-4614d44aed7b/unable-to-access-owa-or-ecp-something-went-wrong-or-500-unexpected-error?forum=exchangesvrclients

    I encountered this problem too and your help has been invaluable to me !

    Have a nice day

    • Hi Neyrim,

      thank you very much for your very nice comment :). I’m glad it helped you.
      you are absolutely right about your remarks. I have to still update this blog.

      Have a nice day too and sorry for late response.

  2. I tried to upgrade Exchange 2010 to 2013 (from a serveur 2k8 SBS to a 2012STD), and got this fuc…ing HTTP 500 when launching Administration console.
    Event viewer was saying SYSTEM doesn’t have token serialization permission (for owa, autodiscover…)
    I solved like this : Get-ClientAccessServer | Add-ADPermission -AccessRights ExtendedRight -ExtendedRights “ms-Exch-EPI-Token-Serialization”, “ms-Exch-EPI-Impersonation” -User SYSTEM
    It rulezzz now 😉

    • check out the following. I forgot to put the complete address:
      CN=Client Access,CN=”your organization name”,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=”domain”

  3. I’m aware this article is a little old and refers to CU3 environments. But I’m out looking for answers and advice as I believe we’re facing a very similar issue.

    In December 2014, we migrated from an Exchange 2013 Hosted environment to Office 365 using an on-prem Exchange 2013 Hybrid server.

    We had applied CU6 across all the mailbox servers and migrated all the mailboxes successfully.
    Now it’s January 16th. The Hosted Exchange servers were decommissioned 2 weeks ago along with an offsite DC which has been decommissioned a few days ago.

    What we experienced just yesterday was that all the admin accounts we setup to access our on-prem Hybrid server could not login and the site page gave a HTTP 500 error and landed at the /owa/auth.owa page.

    The only way we could access ECP was to create another admin account which worked.

    Question (finally): Does anyone out there know if the above instructions work for CU6 servers?
    After some investigation, CU7 was released on December 9, 2014 (1 week before our migration).

    Take a look at Article ID: 3008438

    User who is trying to Log on to Exchange Admin Console is logged in to OWA instead
    http://support.microsoft.com/kb/3008438

    Would this be the same scenario?

    By the way – this is the best site explaining this issue I’ve found so far.
    I’m just surprised it’s taken MS so darn long to address it.

    • Hi Aka Cagen,

      sorry that didn’t reply you asap due to my work :(.

      This should also applies for CU6. CU is a fresh installation, but the schema values for the canary data remains same. So you can try this out.
      If not just let me know to try to help you out.

      Kind regards

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s