Pulumi Uses Real Programming Languages to Enforce Cloud Best Practices

The idea of infrastructure-as-code may be the foundation of DevOps but what startup Pulumi promises is more like infrastructure-as-software.

“What we’re doing is taking all the things we know about software and software engineering, and applying them to infrastructure,” Pulumi founder and CEO Joe Duffy told the New Stack. That gets you all the resources that developers rely on for productivity; something that ops-focused technologies like Kubernetes and even many cloud services have been lacking.

“Existing infrastructure as code technologies that people are using like ARM templates and CloudFormation and Terraform and Kubernetes and YAML and Helm charts are not using programming languages, so you don’t get access to things like IDEs, test frameworks, static analysis or refactoring in your IDE,” Duffy points out. When using Pulumi, the user writes code in IDE, so they get a statement completion; if they make a mistake they get the red squiggle. As they hover, they get the documentation.

The rich ecosystem of tools available for JavaScript, TypeScript, Python and Go go beyond the syntax highlighting and linting that’s available for YAML. Using what Duffy calls “real” programming languages also means you can take advantage of familiar programming concepts of abstraction, encapsulation and componentization. The Pulumi project that describes the infrastructure a developer wants is far more concise than the equivalent JSON or YAML.

“Real languages give you the ability to build bigger things out of smaller; we can leverage abstraction and encapsulate details that don’t matter. Our biggest job when we go into new customers is deleting big gobs of JSON and YAML and DSLs. Often 2,000 lines of YAML evaporates into three or four lines of JavaScript or Python. We’re the YAML dustbusters,” he joked.

With Pulumi’s new Crosswalk framework for AWS, those few lines of code will spin up dozens of resources that developers would usually have to configure by hand, even for simple tasks like running a Dockerized application; depending on your level of experience that’s either daunting or tedious, and as a manual process it’s error-prone.

Spinning up a VPC and using it in an EKS cluster works just like you’d expect, because they’re just programming language abstractions. These high-level concepts map more to the developer’s mental model, Duffy said. “As a result, you can go from container into production in a few lines of code — and be confident that it’s using well-architected practices for AWS by default. You don’t have to go read dozens of white papers to figure out how to make it secure; it just is secure by default.”

Pulumi can make serverless feel much more like event-driven programming, Duffy claims. “Which today it does not, because you usually have a 1-to-1 ratio of YAML to actual code. Whereas here, I’m staying in my editor and just writing code and I’m thinking about S3 events as if they were actual events.” And it simplifies using advances services like AWS API Gateway, which he calls “super-powerful — it’s infinite scale, you only pay for what you use — but incredibly difficult to use today.”

Pulumi CEO Cuts Through the Chafe, Describes Your Real Relationship with Kubernetes

“With Pulumi, the code looks like an ExpressJS app; it’s much more approachable to get up and running with APIs. We’ve made it easy to add dashboards and metrics and alarms, like when my P95 latency goes in the tank, ping this Slack for me,” Duffy said.

The promise of developer productivity is something we hear from a lot of providers. It’s what Google is trying to achieve with Cloud Code and Cloud Run, and what Microsoft is hoping KEDA and Osiris will bring to event-driven programming on Kubernetes; not forcing developers to deal with the operational infrastructure when what they want to think about is their code.

But Pulumi is far more ambitious both in the platforms it’s targeting and the help it promises users.

Code Checking

Not only does Pulumi let developers write infrastructure deployment as code that feels familiar; it also checks that code. That’s a step up even from PowerShell — which Duffy classes as a “real” programming language and expects to support in the future. “.Net is on the roadmap and around that, we will most likely offer PowerShell,” he suggested.

“With the existing PowerShell SDKs, you’re basically open coding stuff and it’s kind of messy. If something fails you’re left in an indeterminate state. Say there are network timeouts or you accidentally ctrl-c your script you’re left in this indeterminate state and you need to figure out how to repair things and now you’re going to manually repair a step. And the API calls are not audited, so you don’t know why people change something when they’re running scripts,” Duffy said.

Even the free open source Pulumi SDK uses the Pulumi cloud service (which has a free tier) to do statement management; the paid tiers for commercial use add policy control and additional secrets and identity management.

“The programmer describes to Pulumi what infrastructure they need, and Pulumi understands what they already have and how to make their desired infrastructure come into existence because it’s controlling all the management infrastructure, it’s doing complete auditing of everything. It integrates into your CI/CD very easily. We do transactional checkpointing so it recovers from failure in the event that failure happens,” Duffy said.

Because of that state management, Pulumi can preview the changes an update will make to the system and predict the impact, he explained. “It shows you a plan. ‘If you deploy this, we’re going to delete these resources and update those resources and oh, by the way, there might be some downtime as a result of it.’ So, you actually understand the semantics of the update before you perform it. That’s one thing you don’t get if you open code scripts; there’s no way to tell what the script is going to do until you run it, at which point it might be too late.”

The policy controls give engineering leaders and operations teams confidence, he suggests. “They want to empower their developers to deploy containers or serverless functions or application-level services, but they want them to do it with guardrails. They don’t want them to go introducing security problems. With Pulumi they can develop their own best practices. If certain people aren’t supposed to make certain kinds of changes you enforce those with role-based access control. If I want to make sure my team doesn’t deploy an unencrypted S3 bucket; that kind of policy can also be put in there.”

The emphasis on “real” programming languages doesn’t mean that Pulumi is only for developers. “Before we launched, I was concerned that we would have something that would appeal to developers but either alienated or didn’t appeal to infrastructure engineers, but it’s actually been quite the opposite,” noted Duffy.

“One thing that works in our favor is Chef and Puppet and Ansible use general purpose programming languages. It’s not quite the same because it’s not for provisioning, it’s for configuration, but that laid the foundation. Most of them are using Python for scripts or PowerShell if they’re in Azure, so a Lot of them are familiar with scripting.” And whether or not they realize what they’re missing, Duffy maintains “they’re hitting their heads against the wall all the time because they don’t have real programming languages.”

Keeping such a wide spread of users satisfied means keeping up with the speed of cloud updates and new releases. Some of that comes from the community and partners. The best patterns and practices enshrined in Crosswalk are based on the approaches the company has seen to be successful for customers, as well as from white papers and guidance from AWS. Duffy expects community contributions will help scale that in the future.

Pulumi recently added support for Digital Ocean and integration for Codefresh, and Duffy told us Crosswalk is likely to support Azure, GCP and Kubernetes “in the not too distant future.” It’s even possible that a subset of Pulumi that’s focused purely on cloud-native technologies could be carved out and donated to the CNCF, but Pulumi proper will continue to support the breadth of services on Azure, AWS and GCP.

Impressively, new services on those clouds are supported within 48 hours of launch (or sooner if Pulumi gets access before launch); that easier for Kubernetes, Azure and GCP than for AWS because they use OpenAPI specifications that allow the small startup team to autogenerate much of the code required to support them. “We can often beat the Kubernetes release out the door once they tag the git commit that’s going to be released,” Duffy boasts.

Pulumi is a sponsor of The New Stack.

Feature image (left to right): Luke Hoban, Pulumi chief technology officer; Eric Rudder, Pulumi president, chief operating officer and executive chairman; Joe Duffy, Pulumi CEO.