User Impact
Users may have received repeated credential prompts within the Outlook desktop client.
This issue only impacted customers using basic authentication. Customers were able to use Modern Authentication to mitigate impact for affected users; however, this process may have required several hours to take effect for some customers.
For some customers who have disabled Modern Authentication, there was a secondary issue that was causing the client to attempt to use Modern Authentication regardless of the setting.
Scope of Impact
This issue affected a subset of customers and users who were connecting to the service using basic authentication and utilized service-based search or a Focused inbox.
Incident Start Date and Time
Monday, May 4, 2020, at 7:00 AM UTC
Incident End Date and Time
Monday, May 11, 2020, at 5:05 AM UTC
Root Cause
A recent update to the Exchange Online service contained a code issue that caused repeated credential prompts for basic authentication users.
Glossary
Representation State Transfer (REST) – These APIs are service endpoints that support sets of HTTP operations, which provide, create, retrieve, update, or delete access to the service’s resources. More information can be found here - https://docs.microsoft.com/en-us/rest/api/azure/.
OAuth – Modern Authentication, an authentication protocol. More information can be found here - https://docs.microsoft.com/en-us/azure/active-directory/develop/active-directory-v2-protocols.
Actions Taken (All times UTC)
Tuesday, May 5
4:00 AM – Telemetry indicates that this was when the deployment which triggered the regression was deployed. Support cases indicate credential prompts earlier than that, however, they may be isolated and not related to the regression exposed by this update.
1:43 PM – We received reports that customers were seeing credential prompts when trying to access Exchange Online using the Outlook client. We started an investigation, but were unable to gather meaningful reproduction data.
Wednesday, May 6
3:28 PM – We determined that requests using OAuth 2.0 (Modern Authentication) were returning 401 errors, but that OAuth 2.0 wasn’t enabled.
6:50 PM – We concluded that the problem manifested in Representational State Transfer (REST). This service doesn’t support OAuth 2.0; with the Outlook client trying to use OAuth, credential prompts will appear.
7:20 PM – We reviewed recent changes to see if this may have disabled OAuth. We also discovered that this affects any user who used a service-based search or a Focused Inbox. Users with modern authentication wouldn’t have been affected.
11:38 PM – We found that the regression was exposed by a service update for Exchange Online. When a user makes an authentication request, the service is generating a value which is not correctly parsed, therefore not correctly validating OAuth 2.0 as expected. At this stage, the problematic change had reached 45% of customers worldwide. We stopped the change from reaching any more of the cloud, and started developing a fix.
Thursday, May 7
12:57 AM – We made sure that the fix was enabled in all build versions and updates, so that subsequent service updates would be streamlined with the fix, and prevent all future credential prompts. To make sure that the fix doesn’t impact the rest of the service, engineers estimated this would take approximately three days to complete development, testing and rollout.
2:21 AM – We finished developing the fix and started testing.
3:07 PM – We completed testing and prepared the fix for deployment.
7:39 PM – We started deploying the fix.
Friday, May 8
4:02 PM – The fix reached 33% completion.
Saturday, May 9
6:01 PM – The fix reached 86% completion.
Sunday, May 10
2:41 AM – The fix reached 92% completion.
Monday, May 11
5:05 AM – We confirmed that the fix had completed and declared the incident resolved.
Microsoft Next Steps
Findings
|
Action
|
Completion Date
|
We're reviewing our update and validation procedures to prevent similar issues from reoccurring and to identify issues before initiating deployment.
|
June 2020
| |
We're reviewing our deployment and patching procedures to more quickly identify the source of impact and mitigate impact in a more timely fashion.
| ||
We’re adding extra validation steps within our code to prevent this issue from happening again
|
0 comments:
Post a Comment