Exchange 2013 OWA & ECP failure after deploying CU3

Posted on by
10

Dear gents, this site is still under construction. It will updated soon.

Acronym:
CU                         Cumulative Update
UR/RP                   Update Rollup/Rollup Pack
Ex2K13                  Exchange 2013
KB                          Knowledge base
AD                          Active Directory
URL                       Uniform Resource Locator
HTTP                     HyperText Transfer Protocol
HA                          High Availability

CU3 [Cumulative Update 3 for Exchange Server 2013 (KB2892464)] was released in November 2013. In this article, the content of the update will not be described. A complete list of customer-reported issues resolved in Exchange Server 2013 CU3 can be found in Knowledge Base Article KB 2892464.

With any CU, RP or other update, there may sometimes be difficulties after installation. Maybe this sounds familiar to some administrators out there.

Some issues may occur:

  • Broken virtual directories
  • Powershell shortcuts missing
  • Powershell dll’s missing
  • References missing in registry
  • ASP.net not correctly updated or broken
  • OWA, ECP or ActiveSync not working
  • Certificate problems
  • Etc …

In the case of the Exchange 2013 CU3 and above, the following functionalities stopped working.

  • OWA
  • ECP

Deployment Guide CU3 and above

CU deployment is equal to a full exchange server installation. There is a good  blog, here, on how to install/deploy this update.

If you prefer scripting or don’t want to loose time typing, use the following script to put the servers in maintenance mode.

Symptoms

After deploying Exchange 2013 CU3 and above in DAG or non DAG mode, the users cannot access OWA or ECP although outlook-anywhere is working correctly.

pic2

pic1

Errors:

I’ve noticed in some environments an absence of error or log file entry (unless we couldn’t find it 🙂 ).

In some cases, you’ll find this kind of event ID.
Event ID: 4 & 1309
source: ASP.Net

Or in OWA/ECP = error 500

pic5

pic6

Note: normally you should also see an error message in OWA indicated by, …canary…,  .

We tried everything to solve this problem without success.

  • Updated only 1 node to CU3, the users on the PAM server couldn’t access OWA/ECP.
  • Activated the updated node, the users couldn’t still access OWA/ECP.
  • The 2nd node updated to CU3, with the  same problems
  • The DB activated on the 2nd node, with the  same problems
  • Created new DB’s on both DAG nodes, moved some users, with the same problems
  • Created new users’ and mailboxes on old and new DB’s, with the  same problems
  • Created new virtual directories, with the  same problems
  • Verified IIS settings, …
  • Installed new Exchange 2013 server with CU3, with the  same problems

Cause

Web site protection: URL rewriting, canary URLs to improve security

The main reason why it is failing is because of a “canary” token-mismatch between client and server.

The canary attributes are created during the first Exchange 2013 schema preparation.

Basically a canary is a secret token between the client and server in OWA, ECP or other web services which get stored in the cookie collection of your browser and then get submitted with the various requests that your browser sends. For any request, the value of the GUID stored in the URL is compared to the one stored in session state. If they don’t match, or if the GUID is missing from the URL, the request is considered malicious and it is blocked, the originating IP address is also logged. Eventually the user will get an http error message 500: Internal Server Error, and the server encounters an unexpected condition which prevents it from fulfilling the request.

AD schema: Exchange canary attributes

 pic7

  • ms-Exch-Canary-Data-0
  • ms-Exch-Canary-Data-1
  • ms-Exch-Canary-Data-2

To be frank, I’m not 100% sure due to which value(s), attribute(s) or service(s) the token verification is failing.

The only valuable attribute what I can see in RTM, CU2 and CU3 (still a guess) that can cause a canary failure, but cannot find detailed information, is the attribute below:

  • Global catalog attributes are added by Exchange 2013 RTM: ms-Exch-Sync-Cookie
  • Attributes modified in Exchange 2013 CU2: ms-Exch-Sync-Cookie
  • Attributes modified in Exchange 2013 CU3: ms-Exch-Coexistence-Secure-Mail-Certificate-Thumbprintms-Exch-Sync-Cookie

Is this attribute the cause of canary failure on OWA and ECP? We have to address to MS to be sure.

Scenarios

Note: the exactly point of failure has to be validated by MS. I guess the failure is on 6 or/and 7 due to modified attribute above.

Working scenario:

pic3

  1. Get OWA
  2. Username and password
  3. If login successfully
  4. Session ID will be created by the server
  5. A cookie will send to the user with session id
  6. Server creates shared secret token (canary)
  7. The canary & session ID are saved in session store state
  8. The canary is rewritten into URL and send to the user
  9. Browser will open the requested page

Not working scenario:

pic4

  1. Get OWA
  2. Username and password
  3. If login successfully
  4. Session ID will be created by the server
  5. A cookie will send to the user with session id
  6. Server cannot create shared secret token (canary)
  7. (or) The canary & session ID aren’t saved in session store state
  8. The canary is not rewritten into URL and send to the user
  9. Browser will return error message 500

Resolution

Note: AD backup is required before taking actions

Applies to:  Exchange 2013 all CU’s

Featured image

  1. Open ADSIEDIT.msc
  2. go to CN=Client Access,CN=”your organization name”,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=”domain”
  3. right click “CN=your Client Access Server” or “CN=your single -or multi-role server(s)” and open properties
  4. scroll down until you find msExchCanaryData0, msExchCanaryData1, msExchCanaryData2 and don’t forget the subfolders, there are also canary values stored
  5. Copy all of these values to a text file with their attribute names and paths.
  6. Once copied and saved, erase all these values
  7. Logon on the CAS server(s) or multi-role server
  8. Open IIS management
  9. Go to the application pools
  10. Right click on “MSExchangeOWAAppPool” and click on recycle
  11. Note: during a recycle the established sessions will be lost, but in this case no worries.
  12. Cmd: appcmd recycle apppool /apppool.name:”MSExchangeOWAAppPool”
  13. If single role exchange architecture, reboot all mailbox servers with CU3 deployed
  14. If multi–role, just reboot the server and skip recycling.

Cheers,

Kyn