Employee-facing Mutual TLS
Armen Tashjian | Security Engineer, Corporate Security
This blog article is the second part of our recently released blog: Enforcing Device AuthN & Compliance at Pinterest.
Intro
As part of our device authentication and compliance initiative, Pinterest has implemented employee-facing mutual TLS with a custom identity provider in a way that results in a positive user experience.
You may have heard of, or experienced first hand, some unpleasant behavior while attempting to authenticate with a certificate within a browser or application. Even the Wikipedia page for mutual TLS mentions that mTLS is a “..less user-friendly experience, [and] it’s rarely used in end-user applications…”.
At Pinterest, we needed to use Mutual TLS as part of our employee SSO authentication, using a custom identity provider. This means that we needed to support authentication across all major platforms, as well as from within browsers and native applications.
In this blog post, we’ll talk about some of the changes that we’ve made to ensure that user-facing mTLS is a seamless experience for our employees.
Automatic Browser Certificate Selection
In order to make the authentication experience seamless on macOS or Windows platforms, we have deployed a policy to automatically select the correct client certificate on behalf of a user, with the AutoSelectCertificateForUrls Chrome policy. This results in no certificate prompt for end users. A similar policy exists for other browsers as well.
Unfortunately, similar policies cannot be implemented on Android/iOS.
Defeating the SSLClientAuthCache — Re-prompting for a Certificate
A notable pain point that we attempted to mitigate with mTLS-based auth is related to the user experience when a certificate prompt is accidentally closed by a user, or if an incorrect certificate is selected. The only way for a user to be “re-prompted” for a certificate is to restart the browser.
While forcing a browser restart may be an acceptable solution for some on a Windows/macOS platform, the consequences for making an incorrect decision in a native application on iOS or Android is particularly terrible.
Note that even restarting the native application doesn’t resolve the issue in the example below.
The cache responsible for this behavior on Chromium-based browsers is the SSLClientAuthCache, which is described as:
A simple cache structure to store SSL client certificate decisions. Provides lookup, insertion, and deletion of entries based on a server’s host and port.
A simplified representation of this cache is below:
It’s also apparent why cancelling a certificate prompt does not cause a re-prompt, as Chromium-based browsers see a “cancelled” certificate prompt as an intentional action:
The desired certificate may be NULL, which indicates a preference to not send any certificate to |server|.
In the description of the SSLClientAuthCache above, you might have noticed that the cache performs lookups “..of entries based on a server’s host and port.” This suggests that it would be possible to create a new entry to this table by changing either the port or the hostname of the server that a client is interacting with.
Since we control the edge infrastructure that clients interact with, we can take advantage of this behavior to defeat the SSLClientAuthCache with a server side change. We can simply redirect users who have not passed a valid certificate to a random subdomain, which then triggers the user’s browser to reprompt for a certificate. If the user still does not present a certificate, they are then redirected to an error page where they can try again if necessary.
In the GIF below, we demonstrate our mTLS implementation with our custom identity provider. Note that even within a native application, canceling the certificate prompt can be remedied in an intuitive way.
Below is the routing logic responsible for this as implemented in our edge infrastructure (Envoy), which can be replicated in other proxy/web server implementations as well.
Disabling HTTP/2
In order to properly trigger a certificate prompt for random subdomains, we also needed to disable HTTP/2. The reason for this is related to the connection reuse properties of HTTP/2, described in section 9.1.1 of the HTTP/2 RFC.
Although the RFC references that, “A server that does not wish clients to reuse connections can indicate that it is not authoritative for a request by sending a 421 (Misdirected Request) status code,” we found that Envoy does not adhere to the RFC in this respect, and 421 responses are not sent to clients.
In any case, even if Envoy did adhere to the RFC, expecting clients to receive and handle the 421 responses unnecessarily complicates our implementation, so we found that simply disabling HTTP/2 for communications with our custom identity provider was the best solution.
Distinguished Names of CAs
Another server side change that can improve the user experience is properly configuring the list of distinguished names of acceptable CAs, which is described in the Certificate Request of the TLS 1.2 RFC. Many client applications (i.e. browsers) will attempt to present users only with client certificates that have been signed by one of the CAs that are present on this list.
As mentioned in the RFC, if the list is empty, the client may send any valid certificate. Your browser will then prompt you to select from all of the certificates that you might have available, even if they won’t be accepted by the server. This results in a particularly bad (and avoidable) experience for users, as they will be prompted to select from a list of certificates that the server will end up rejecting.
Issues
WebView Compatibility
Since we are implementing mTLS authentication as part of our Okta SSO authentication flow, native applications need to be able to redirect users to a browser capable of accessing the keychain/certificate store.
If application developers were following best practices for federated authentication, this would be a non-issue. Unfortunately, we have run into a significant number of native applications for “enterprise” tools, which continue to prompt users to authenticate to Okta from within a WebView, as opposed to using appropriate alternatives such as Chrome Custom Tabs for Android, and ASWebAuthenticationSession for iOS/macOS.
Aside from the compatibility issues that WebViews present for both FIDO2 and mTLS, there are real security issues that WebViews present, including phishing and SSO session hijacking.
In the technical requirements that we share with prospective vendors, we cover the risks that WebView usage presents in more detail, as well as the correct implementations that we require application developers to follow in order for mTLS and FIDO2 to work correctly.
iOS Non-Safari Users
On iOS, certificates in the system keychain cannot be accessed by Chrome. This presents an issue for some of our users who have Chrome installed as a default browser on their iOS devices.
To make matters worse, there are some native applications that will open the default browser to authenticate, as opposed to using something like a SFSafariViewController or ASWebAuthenticationSession, which means that users with Chrome as a default browser simply cannot use those apps.
Our guidance has been to only use Safari as the default browser on iOS.
Android Work Profile
Although from a security perspective, it’s desirable that provisioned certificates are accessible only by applications in a user’s work profile, this is something that might cause friction from a UX perspective. It is not immediately clear to a user why an application they are trying to access in their Personal profile is not able to access the certificate that only exists in the Work profile keychain.
We do surface this as a troubleshooting step in the error message presented to users on Android devices (i.e. “make sure you’re using your work profile apps”), but it’s something that can result in help desk tickets for resolution.
Conclusion
Since implementing our Mutual TLS-based solution for SSO about 3 months ago, we have a seen an average of 13k weekly authentications. The average number of related helpdesk tickets are less than five.
For those who have shied away from using mTLS for user-facing authentication, we highly recommend considering it as an option.
Many thanks to our partners in Pinterest’s Traffic Engineering team for helping to implement this solution.
For any thoughts or feedback, feel free to reach out to zuul[at]pinterest.com
To learn more about engineering at Pinterest, check out the rest of our Engineering Blog and visit our Pinterest Labs site. To explore life at Pinterest, visit our Careers page.
