After some advice please! The org I work for is a very Engineering heavy workforce and Zscaler is a constant disruption to their development work (either due to SSL inspection or slow performance).
For now, we have bypassed SSL inspection completely for Engineers which of course this is a massive gap but the best approach we could take
Is there any similar orgs here and if so, how has Zscaler worked for you around this issue?
EDIT: Thanks for all your advice, it was really helpful and got some good ideas here I’ll take away and look into. Now time for the battle vs engineers… wish me luck!
Does engineering mean software development? I know a dev org that documented how to get the Zscaler certificate trusted in all of their software components on a wiki the dev team ran. Short answer for their security team was if they’re technical enough to write code, they’re technical enough to figure out how to make their app trust a certificate
A big part of this is something your security org needs to drive from the top down. The biggest problem I’ve seen with dev teams is that they often have very few standards, and are all building and deploying in their own little worlds, often without thinking about security, or the bigger picture.
Companies that do this successfully, set security standards (like importing the intermediate certs) and other settings, so that you don’t run into these issues. They aren’t unique to Zscaler either, you would have the same issue with Palo, Netskope, and so on.
It’s not a technical issue that you can solve, it’s a political one within your company. The biggest prob you have now, is once you’ve your all these bypasses in place, it’s very hard to undo them.
Bypassing completely is absurd. There are maybe 5-6 categories that are likely to cause friction with technical sites.
Try to find a pilot group that can help to identify exceptions. Give these users the ability to disable ZIA when they hit friction so it doesn’t disrupt their workflow - with the understanding that they report the url to you.
Make a custom category to bucket these exceptions for the engineering department.
The biggest improvement we found to performance have been to make sure none of the employees needed a VPN while using Zscaler and Whitelist everything about it in Windows Defender.
Turns out Zscaler can write a lot of logs if you let it, we found massive improvements in lowering the log levels and making sure the anti-malware/virus/everything was not scanning those logs.
I’d set up an internal permalink of some kind for a PEM or CRT version of the root certificate, give the devs the Zscaler link here, and let them self-service their environments.
You need pilot users, a workshop where you add the certs to the store and fix issues and SLT support so the devs that are your pilot are on board. I had to spend 3 days in person with a customers dev ops team after they failed to deploy Zscaler to them. They tested everything and we became friends, I had the group CISO on teams green lighting changes and any completely needed bypasses. I also had to break the “not my job” boundaries helping add ZS cert to terraform scripts etc. (I know this was easier because I am a consultant) I work for an MSSP your issue isn’t unique, Zscaler needs to be deployed slowly and carefully with these groups.
I look after a 100k+ prod user environment with ZIA with a seperate tenant for about 2.5k software engineers.
In both tenants, we us custom root certs from our respective internal CAs (prod and dev) and just ensures that the root cert is installed on all devices.
Devs can then export the Root cert from their local device and use it in whatever repo / cert store they need for their app.
We’re pub sec so security is massive for us. We have slightly more relaxed policies for Devs but SSL inspection underpins almost every useful control in ZIA. Unless you can’t get around it (e.g. Cert pinning) then I suggest inspection is used as much as possible.
Zscaler just released to some proper guidelines for ssl inspection. Not sure if it’s available in the internet. Contact your TAM and see if you can get a copy of it. Very comprehensive and useful information with details of policy implementation on how to build and bypass.
There are a lot of good tips here, but in general I’ve always seen issues with wanting to do SSL inspection for software engineering-heavy organizations. It ends up breaking all sorts of stuff and you’re left either constantly playing whack-a-mole with bypass exceptions or you turn it off completely for engineers. This is not unique to zscaler, but any in-line proxy that break SSL.
I’ve seen deployments where they run ZIA in parallel with something like Twingate (developer friendly VPN) for the eng team so they get private access to build servers, VPCs, etc, while still inspecting the public internet traffic with ZIA. This can work pretty well if that’s the problem causing the engineers issues.
I get why people say bypassing everything is absurd and “why can engineers just figure it out” but political battles are hard to fight. And if you’re in an engineering-heavy org, you’re always going to lose that fight to the engineers.
My job aid redacted if it helps. Small number of developers, but this has all worked fine:
Overview
Zscaler traffic is intercepted and decrypted. A managed workstation has the Zscaler root certificate installed on the OS certificate store through the installation of the Zscaler client. Without that root certificate errors would occur when accessing a site protected by SSL/TLS encryption schemes.
XXXXXXX software developers may have tools installed that use their own certificate store. The Zscaler root certificate may have to be installed in those individual application certificate stores for them to interact with Internet resources correctly. A common tool in this class is Git or Git related tools for interacting with GitHub. Another common tool is NPM for JavaScript development.
Certificate Errors
Obtain the certificate and apply:
The developer can find the Zscaler root certificate here:
xxxxxxxxxxxx
The developer would be responsible for updating and maintaining their hosts. They are responsible for their tools and the methodology to add a cert to the application specific cert store. The developer may have to change the format of the cert according to the tools’ instructions.
The certificate expiration date: xxxxxxxxxx
Zscaler help on the subject, gives several specific tools examples including Git, NPM, Java and others:
The solution is either import the Zscaler root certificate into the tool’s certificate store or in some cases the tool may have a variable to point the tool directly at the Zscaler Certificate.
File Download Errors
Once the certificate is trusted there still may be errors downloading packages. That may manifest as 403 errors or connection failures or time outs. This may indicate that Zscaler has quarantined the file for scanning or even blocked it as malicious. Start a ticket with the Service Desk for these issues. On the ticket please provide:
Name of person experiencing the issue
Date and time the issue was experienced
Destination Site name, URL or IP address that failed
We wrote an internal tool that puts the zscaler certificate in the correct places for a lot of tools. Everyone internally can contribute or request support for a new tool to be supported.