CircleCI CTO on How to Quickly Recover from a Malicious Hack
Just as everyone was heading out to the New Year’s holidays last year, CircleCI CTO Rob Zuber got a surprise of a most unwelcome sort. A customer alerted CircleCI to suspicious GitHub OAuth activity. Although the scope of the attack appeared limited, there was still no telling if other customers of the DevOps-friendly continuous integration and continuous delivery platform were impacted.
This notification kicked off a deeper review by CircleCI’s security team with GitHub, and they rotated all GitHub OAuth tokens on behalf of their customers. On January 4, the company also made the difficult but necessary decision to alert customers of this “security instance,” asking them to immediately rotate any and all stored secrets and review internal logs for any unauthorized access.
In this latest episode of The New Stack Makers podcast, we discuss with Zuber the attack and how CircleCI responded. We also talk about what other companies should do to avoid the same situation, and what to do should it happen again.
“I think it’s a tough decision. It shouldn’t be a tough decision. But it is a tough decision, I think when something like this happens, it feels like you’re putting yourself at risk,” Zuber said, on making the decision on alerting all customers about an attack on a single client. “But the reality of the situation is for an event of the scope, and size and impact on our customers, there weren’t a lot of choices. What the best thing that we could do for our customers was what drove all of our decision making.”
After the first disclosure, CircleCI followed up with customers offering a more detailed examination of what took place. This transparency was important for the company, and CircleCI openly published details on every public platform it could.
“It was important to us to make sure that every customer could check their own exposure,” Zuber said. “There was not really a model for us to figure out every single customer that was impacted, only those that could be, which is the full scope of our customer base, effectively, then informing those customers [on] what impact would look like to the best of our understanding.”
It turned out that the attack was just on the one customer. Nonetheless, CircleCI did make some changes in its tooling, and in scaling down the scope in which access permissions are granted. In general, the company has started moving away from “a model where you would have secrets with us and into one where you’re doing a secure token exchange on the fly as part of the build,” Zuber said.
And this is where the industry needs to head, Zuber said. “We can always enhance how we’re managing things ourselves in terms of our delivery, and in terms of the system that we’re running. But if the exposure inside that system is the short-lived tightly-scoped tokens, then the overall sort of potential [for a breach] is massively reduced in the future,” he said.
“The place that we’re really interested in is not just changing our product, but helping the industry move forward.”
