The OpenStack community has released its 27th version of the software, circling all the way back to the beginning of the alphabet. Due to its passionate and active contributor base, OpenStack continues to be one of the top five most active open source projects. Organizations around the globe, spanning almost every industry, have embraced OpenStack, reaching 40 million cores of compute in production. Within this footprint, adoption specifically among OpenStack-powered public clouds now spans over 300 data centers worldwide.

In addition to OpenStack, the OpenInfra Foundation has replicated its model for hosting open source projects including Kata Containers, StarlingX and Zuul. This model is now readily available for any organization that wants to leverage the Four Opens and three forces to build a sustainable open source project within the infrastructure layer.

The OpenInfra Summit Vancouver on June 13-15, is a great opportunity to get involved in the OpenStack community while collaborating more closely with other OpenInfra projects and learn from the world’s biggest users.

Red Hat OpenShift is for innovation without limitation. Bring big ideas to life with the hybrid cloud platform open to any app, team, or infrastructure.

Learn More

The latest from Red Hat

OpenStack Is More Reliable and Stable Than Ever

As the OpenStack software platform has matured, there has been a notable emphasis on reliability and stability. Many features and enhancements have been introduced to ensure a smoother and more robust experience. These improvements include the implementation of a new “skip level upgrade release process” cadence, which began with the Antelope release in March 2023.

One significant aspect of OpenStack’s evolution is the increased emphasis on thorough testing. More extensive testing procedures are now in place, ensuring that the platform is attentively examined for potential issues and vulnerabilities.

Another recent focus for the upstream community has been removing under-maintained services and features to allow for a more focused and efficient system, eliminating unnecessary components that may hinder the reliability and stability of OpenStack.

OpenStack also places a strong emphasis on interoperability. Integration and collaboration efforts with other popular open source components such as GNU/Linux, Kubernetes, Open vSwitch, Ceph and Ansible have been prioritized. These initiatives promote compatibility and interaction between different software systems, which has enhanced overall reliability.

In contrast to the past focus on vendor-specific drivers and niche features, OpenStack now prioritizes contributions to general functionality. For example, developing a unified client/SDK offers a standardized and consistent experience across the platform. This shift promotes stability and reliability by focusing on core functionalities that benefit all users.

As OpenStack continues to mature, these various measures and initiatives demonstrate a strong commitment to reliability, stability and long-term success.

Flexible Support for a Variety of Workload Models

OpenStack is a powerful and versatile cloud computing platform that offers flexible support for various workload models. One of its standout features is the ability to work closely with hardware, enabling users to harness the full potential of their systems. For instance, the Ironic bare metal systems deployment and lifecycle management tool and the Nova bare metal driver provide seamless integration of on-demand physical server access into a full-featured OpenStack deployment.

To ensure long-term sustainability, OpenStack’s capabilities are continuously tested on architectures like ARM/AArch64 and have included support for other unconventional processor architectures like PowerPC. It also offers advanced scheduling capabilities like PCI passthrough, CPU pinning, and coordination and life cycle management of peripherals like accelerators, graphics processing units (GPUs), data processing units (DPUs) and field-programmable gate arrays (FPGAs). Moreover, OpenStack has tighter integration with the container ecosystem, with Magnum as a Kubernetes-certified distribution, Zun enabling individual application containers to be provisioned and manipulated as first-class server objects and Kuryr delivering advanced Neutron network features directly to container processes.

OpenStack also offers solutions for running its services as container workloads, with Kolla and OpenStack-Helm. It has fostered close collaboration with the Kubernetes community, with current and former leadership cross-over between the two projects. OpenStack provides services to facilitate long-lived processes and precious data, such as scheduling policies, data retention, backups and high availability/disaster recovery. Its services facilitate ephemeral and distributed applications with load distribution and multi and hybrid cloud, along with cloud-bursting features. Overall, this is an ideal platform for organizations looking to achieve maximum flexibility and efficiency in their cloud computing environments, with a broad range of tools and features that can support a wide variety of workloads and use cases.

What’s the Story with Security?

Security is a major concern in any computing platform, and the OpenStack community takes this issue very seriously. Over time, the OpenStack contributors have made significant strides in enhancing security through long-term improvement initiatives. Community goals have been set to tackle critical security aspects, such as role-based access control, privilege separation for services, image encryption and Federal Information Processing Standards (FIPS) compliance testing. These efforts demonstrate the community’s commitment to continuously enhancing security features and mitigating potential risks.

One notable achievement is the steady reduction in the volume of reported security vulnerabilities. By actively identifying and addressing security concerns, the community has created a safer environment for cloud deployments.

Additionally, OpenStack has implemented new vulnerability coordination policies that promote transparency and collaboration. These policies not only provide open access to more projects but also mandate clearer publication timelines. By ensuring that vulnerabilities are promptly disclosed and addressed, OpenStack enables users to stay informed and take appropriate actions to protect their systems.

OpenStack’s commitment to security extends beyond its own ecosystem. The platform has been a pioneer in establishing a sustainable vulnerability management process, which has served as a model for many other open source communities. This recognition highlights the effectiveness of OpenStack’s security practices and reinforces its position as a leader in the open source world.

How Can People Contribute and Add to This Project?

The OpenStack community welcomes all individuals and organizations to actively participate and enhance the community by adhering to OpenInfra’s “Four Opens” principles. If you’re interested in joining this collaborative effort, various avenues are available to guide you through the process.

To begin contributing, the project offers Contributor Guides that serve as valuable resources for both individuals and organizations. These guides not only assist with upstream code contributions but also provide insights into non-code contributions. Additionally, they outline opportunities for users and operators to contribute their expertise and insights to the project’s growth.

One way to make a meaningful impact is by volunteering as a mentor for university interns. Sharing your knowledge and experience can help shape the next generation of contributors. Moreover, you can propose efforts for sponsorship through programs like Outreachy, which provides opportunities for individuals from underrepresented backgrounds to contribute to open source projects. Additionally, you can support events such as Open Source Day at the Grace Hopper conference.

For someone who is new and seeking information and advice, the First Contact SIG (Special Interest Group) within the OpenStack community is an excellent starting point. This group’s mission is to provide a place for new contributors, making it a welcoming and inclusive space for those who are just beginning their journey in the project.

If you’re looking to make a more significant impact, consider exploring Upstream Investment Opportunities. These opportunities offer a curated set of suggested investment areas based on the current needs in the OpenStack community along with contact points for who can help you get started.

Overall, the OpenStack project offers a range of avenues for individuals and organizations to contribute and add value to the community. Whether it’s through code or non-code contributions, mentoring, sponsorships or investment opportunities, there are numerous ways to engage and actively participate in the growth and success of the project.

The post What’s Up with OpenStack in 2023 appeared first on The New Stack.

It’s no industry secret that the cloud native segment around Kubernetes has shifted toward hosted Kubernetes providers who build, run and partially manage the Kubernetes infrastructure for organizations. Compared to organizations building and maintaining their own Kubernetes infrastructure, hosted Kubernetes providers allow you to offload a measurable amount of technical complexity so staff can focus on operations and innovation.

Along with the rise of hosted Kubernetes providers, more enterprises are favoring larger Kubernetes distributions from the likes of OpenShift, Rancher, Azure Kubernetes Service (AKS), Amazon Elastic Kubernetes Service (EKS) and others rather than building their own homegrown distribution from the upstream codebase.

These trends are not limited to the Kubernetes platform itself but showcase a general movement toward letting the providers of strong core platform layers do what they do best so companies can focus on the business value that comes from building on top of Kubernetes. This was a chant heard in 2017 to “make Kubernetes boring,” and I think that we are getting there as an ecosystem.

But that was six years ago. What does “boring” look like in 2023 and how do new trends like the rise of Kubernetes operators fit into this picture? There are three ways I think of this when evaluating modern Kubernetes deployments:

I want my organization to build value on top of Kubernetes.

Similar to the mantra of 2017, the “value” we mean here is everything that is built on top on Kubernetes and the infrastructure layers, which has seen substantial progress and evolution from the community over the past six years.

I want Kubernetes to be simple.

Every organization is unique, and roles within your organization may differ depending on not only size, but also Kubernetes maturity. Because of this, skill sets vary, and not everyone has the time or ambition to become an expert. Those who aren’t experts want Kubernetes to be easy so daily tasks aren’t intrusive.

I want Kubernetes to be scalable.

Deployment models for Kubernetes are expanding, and enterprises are taking advantage of using Kubernetes across on-premises, multicloud and hybrid cloud environments. Kubernetes needs to be flexible across these environments while also enabling cluster growth with streamlined scalability as the practice matures.

Dell Technologies (NYSE:DELL) is a unique family of businesses that helps organizations and individuals build their digital future and transform how they work, live and play. The company provides customers with the industry’s broadest and most innovative technology and services portfolio spanning from edge to core to cloud.

Learn More

The latest from Dell Technologies

Building Value on Top of Kubernetes

Once the Kubernetes infrastructure layers are solid for your organization, it’s time to build the “value” on top, whether that is an application end users interact with or a platform layer that adds advanced data services such as observability. Developers need to start somewhere, and this usually consists of finding the right Kubernetes resources for the workload, such as creating deployments, services, jobs, statefulsets, daemonsets, persistent volumes, pod security policies, role-based access control (RBAC) rules, secrets, service accounts and much more.

Managing and tracking all these resources can get quite complicated, and it’s likely that your team doesn’t need to control all these objects but must adhere to resources that affect how applications run. There are cases where this development practice is something that must happen: For instance, if the application you are building is unique to your organization, then the API resources prevent you from having to start from scratch.

However, on the flip side, we see DevOps teams, developers and application owners turning to trusted, prebuilt Kubernetes operators to run, configure and manage common applications so they can focus on the value above these layers.

Operators: Bringing Together Value, Simplicity and Scalability

If you’re not familiar with what a Kubernetes operator is, then I suggest reading the documentation.

Switchboard operator

However, whenever I hear the term “operator,” my mind immediately jumps to a switchboard operator with a massive telephone network in front of them moving wires in and out at a rapid pace while transferring calls.

You may remember them from the pilot of the hit show “Mad Men” or recall the popular saying, “Operator, please hold.”

Much like the way a switchboard operator in the 20th century assisted in the routing and transfer of phone calls, a Kubernetes operator facilitates the deployment, management and ongoing operations of a Kubernetes application. Except instead of having a person move wires behind a telephone switchboard, think of it as a robot who is listening to the inputs and commands and outputting the Kubernetes resources in the appropriate namespaces.

It’s Like a Robot, but without the Attitude

Unlike the switchboard operator, the core tenet of a Kubernetes operator is automation. Automation is a necessity as the community forges ahead with Kubernetes, allowing end users to focus on what matters to them while relying on operators to automate deployments, operations and management of common components in their stack.

There is a community propensity to use trusted operators for applications and not reinvent the wheel when running a particular service on Kubernetes. Take the database landscape’s use of operators as an example.

As seen at KubeCon EU in Amsterdam, the operator pattern has a strong use case for databases because in general; they are a common denominator to many application stacks. Applications may use Postgres or Redis in slightly different ways, but they are common services that need to be installed, configured and managed. Databases on Kubernetes deployed via operator in a trusted way for production is a major win for time-to-value when it comes to DevOps development cycles.

It doesn’t stop at databases, though; operators can be used for all kinds of applications. Operators can be used for almost anything from monitoring and alerting software, to storage integrations, to fully customized applications that may be delivered to internal customers.

It’s great to see the focus move northbound as the Kubernetes ecosystem matures. As end users and organizations are gravitating to hosted Kubernetes and application automation through operators, I’m excited to see the innovations that come next focus on what can be built on top of Kubernetes.

How Do We Use Operators?

Operator frameworks are extremely popular among Dell’s customers, and we are actively working to introduce deeper operator capabilities for our Kubernetes storage capabilities, such as our container storage modules, as well as container storage interface drivers, which are available on OperatorHub.io. Operators are also a key part of our future portfolio offerings and will be integrated into our upcoming user interface for Kubernetes data storage.

The benefits of using operators are straightforward: less time spent on manual processes, more time spent on coding and innovation. If you haven’t started with operators today in your business, I highly suggest exploring the world of Kubernetes operators and seeing how to take advantage of automation to make your life a little easier.

Simple, scalable and adding value on top of Kubernetes.

The post Kubernetes Operators: The Real Reason Your Boss Is Smiling appeared first on The New Stack.

The overwhelming majority of developers in a new survey — 84% — say they’re involved in DevOps activities. But despite this, devs haven’t gotten any faster at making code changes and putting them into production over the past two and a half years.

The increasing complexity of the projects DevOps teams work on may be slowing down developer velocity, suggested the report by SlashData and the CD Foundation, a project of the Linux Foundation.

Among the findings:

• Lead times to restore service after an outage have increased. According to the report, it takes up to a day for 44% of DevOps practitioners to restore services compared to 54% who said the same since data collection began in the third quarter of 2020.
• The time needed to implement code changes has not improved. Thirty-seven percent of DevOps practitioners said their lead time for code changes is less than a week. That’s the same percentage that gave that answer in Q3 2020.
• The proportion of organizations that the researchers define as low performers has increased since Q3 of 2020, while the share of high performers has decreased in that same period. This fits a pattern we have written about previously.

“It is good to see that there is still an increase in the adoption of CD and DevOps practices. However, there are signs that we still have work to do,” said Fatih Degirmenci, executive director of the CD Foundation, in an email response to The New Stack.

“CD and DevOps require organizations to change how they organize themselves, get their teams to embrace the cultural and mindset changes and adopt their product structures,” Degirmenci wrote, adding, “These changes usually take some time to implement and show their effects.”

He also noted the impact of the complexity involved, including “not just how the products are built (e.g. microservices) but also the surrounding environment — from infrastructure to development environments to CD technologies, and so on.”

The new report’s findings are derived from data collected for SlashData’s past six Developer Nation surveys, which reached more than 125,000 respondents from the third quarter of 2020 to the first quarter of 2023.

Use of Self-Hosted CI/CD Tools Declined

The study’s results showed how organizations are using DevOps technologies. Not much has changed since Q1 of 2022, including the average number of tools used, which has held steady at 4.5.

However, some trends emerged:

• The use of self-hosted CI/CD tools has dropped from 32%  in Q1 2022 to 23% in Q1 2023.
• Application monitoring/observability tools saw the second biggest decline in usage, going from 37% to 31% in the same period.
• Few areas of security saw increased use. The largest increase was for application security testing technologies, which increased from 25% to 28% over the past year.

CI/CD tools appear to be a decisive factor in how quickly it takes organizations to restore service: Fifty-eight percent of those that use CI/CD tools can restore within a day, versus 35% of those that don't.

The more CI/CD technologies organizations use, the less time it takes to restore. For example, it takes more than a week to restore services for about 55% of DevOps practitioners, but that steadily drops for each additional tool, going to approximately 20% among those using eight or more tools. But there’s a caveat: the more self-hosted tools developers use, the longer it takes to restore service after an incident.

More than half of organizations that use five or more self-hosted tools were defined as low performers by the researchers, with only 10% of organizations that use that many self-hosted tools are considered high performers.

“An increasing number of tools used having such a strongly negative impact on service restoration time has multiple possible explanations,” the report stated. “However, interoperability issues may be at the centre of many of them. Multiple tools may make it challenging to integrate all of them well, leading to a greater challenge to isolate the service-impacting issue at hand.

“Further, a lack of standardisation between tools may make it more difficult for all tools to work together well.”

As teams choose from among the ever-expanding landscape of DevOps tools, they should consider how well the tools they use play together, Degirmenci suggested.

“One thing organizations could do and benefit from,” he wrote The New Stack, “is including interoperability as a criteria during their technology evaluations, so they can reduce the complexity as well as reach greater flexibility when it comes to adding new tools to their environments.”

The post Is DevOps Tool Complexity Slowing Down Developer Velocity? appeared first on The New Stack.

Jyoti Bansal is the CEO of Harness, a DevOps company that has recently fine-tuned its tagline to “the industry’s first Software Delivery Platform to use AI to simplify your DevOps processes.” Harness isn’t the first company to pivot hard towards AI, of course, but given its focus on continuous delivery, I took the opportunity to interview Bansal about how AI is now used within Harness and by its customers.

I began by asking Bansal what impact generative AI has had on developers and their workflow.

He replied that AI can alleviate the tedious and repetitive tasks involved in the software delivery lifecycle, starting from generating specifications based on existing features, to writing code. Also, he said that AI can automate code reviews, vulnerability testing, bug fixing, and even the creation of CI/CD pipelines for builds and deployments.

“We can easily get 30-40% efficiency in every part of the developer workflow by intelligent use of generative AI,” Bansal said.

How AI Is Used in Harness

Since Harness itself makes heavy use of AI technology, I asked Bansal for details on this.

One key area where AI is utilized, he told me, is in ensuring that code changes do not negatively impact performance, quality, or security. Harness leverages AI models within their continuous delivery pipelines, he explained, which compare code changes against data from monitoring and logging systems like DataDog, Azure Monitor, and Splunk. He said that these AI models can identify any potential issues before code changes are deployed into production in their systems, which enables rapid and reliable deployment pipelines.

Another AI technique employed by Harness is what Bansal called “test intelligence.” This addresses the common challenge of lengthy test execution times, he said. By using AI models, Harness identifies the parts of the code that correlates with specific tests, allowing its developers to optimize the tests that need to be run. Instead of running a large suite of tests for every code change, Harness can determine the specific tests required for a given code change. This significantly reduces test execution time and increases developer productivity.

“We have many other smaller techniques that we have been building to optimize things,” he said.

Bansal also confirmed that it is actively developing generative AI-based approaches to further enhance its platform, but he wouldn’t give any further details about these as yet unannounced features.

Embedded vs. Sold

One of Bansal’s adages about AI is that it should be embedded in B2B products, rather than sold as a separate add-on. In customer support, he explained, AI can be used to generate appropriate responses to customer queries. Similarly, in sales, AI can assist with outreach campaigns to potential buyers.

As for developer products, Bansal noted that customers are increasingly demanding AI capabilities be directly integrated into the products they purchase, rather than relying on separate AI solutions. He said this approach ensures that AI is tailored to specific use cases and domains.

Prompt Engineering

I asked if Harness has any features that help with the new role of “prompt engineering,” which many developers are now adapting to (for example when getting code completion help from GitHub Copilot).

Bansal acknowledged the importance of prompt engineering and what he termed “multi-agent techniques” in the DevOps process. He said that Harness is actively working on incorporating these concepts into their platform, but as yet it has nothing to announce. But his goal, he said, is to create a model that facilitates the developer workflow by utilizing agents for various tasks. So you’d have agents that create code specifications, write code, generate test cases, and perform code testing.

Security

Bansal is also the CEO of a newer company he co-founded, called Traceable (“Intelligent API Security at Enterprise Scale”). Traceable watches and analyzes every API call, ensuring that sensitive data is not inadvertently sent to GPT, APIs, or other endpoints, he explained. It leverages AI technology to do this monitoring and to detect any potential security breaches or data exfiltration attempts.

Traceable’s AI models are built on an API data lake, Bansal said, which allows security teams to identify and address security challenges in real time. He added that the demand for securing generative AI-based traffic is rapidly growing, especially in industries like banking and financial services where data privacy is a significant concern.

Advice for IT Departments and Developers

Bansal advises IT departments in enterprise companies to leverage generative AI capabilities offered by their vendors, rather than attempting to build them in-house. He suggests that companies focus on integrating generative AI in areas that are not their core business — such as developer tools, sales tools, marketing tools, accounting, and HR. By pushing their vendors to incorporate generative AI capabilities into these products, companies can benefit from the advancements without investing significant resources, he said.

However, Jyoti also noted that it is worthwhile for companies to explore the use of generative AI in their core business areas, where it can enhance customer experiences. For example, banks can consider using generative AI bots in online banking, and retailers can explore generative AI-based personal shopping assistants.

As for developers, he emphasized the importance of learning how to use AI effectively and incorporating it into every part of the software delivery lifecycle. By harnessing generative AI, he said, developers can be more efficient and complete tasks in less time.

The post AI Has Become Integral to the Software Delivery Lifecycle appeared first on The New Stack.

It’s at the point where GitOps is getting enough notice that a brief on its principles is appropriate.

Last year, the OpenGitOps community released GitOps Principles 1.0. There’s general support for GitOps and many competing interests in developing GitOps engines, such as with Argo and Flux, two graduate projects from the Cloud Native Computing Foundation. But focusing on principles lets everyone know what GitOps is and, even more so, helps define what it is not.

OpenGitOps defines GitOps as a set of principles for operating and managing software systems, wrote open source software engineer Scott Rigby. “When using GitOps, the Desired State of a system or subsystem is defined declaratively as versioned, immutable data, and the running system’s configuration is continuously derived from this data. These principles were derived from modern software operations but are rooted in pre-existing and widely adopted best practices.”

With DevOps, operations and development teams collaborate on their own chosen tools. GitOps provides a declarative approach and complements DevOps. It allows for application delivery and cluster management. GitOps shares the same concepts as Kubernetes, making it easier for teams already working with it to adapt.

The cdCON + GitOpsCon conference, held this week in Vancouver, BC,  featured a presentation about the GitOps principles by Rigby, and Christian Hernandez, a senior principal product manager with Red Hat.

Here are a few takeaways, from this talk and others at the conference:

Principle #1: GitOps Is Declarative.

A system managed by GitOps must have a desired state expressed declaratively.

GitOps allows for automating security practices, said Eve Ben Ezra, a software engineer with The New York Times, who spoke at the cdCON + GitOps Con event. DevOps encourages collaboration, which means incorporating security into every stage of the software development lifecycle.

The comparison to security practices dovetails with the second principle of GitOps:

Principle #2: GitOps Apps Are Versioned and Immutable

The desired state is stored in a way that enforces immutability, versioning, and complete version history.

The general viewpoint: rollbacks should be simple.

“You go a step further with GitOps providing an auditable trail of all changes to infrastructure and applications,” Ezra said.

Versioning allows organizations to find gaps in their security and also allows for testing and declarative infrastructure, Ben Ezra said. Using tools like the Open Policy Agent, an open source project for establishing authorization policies across cloud native environments, allows for more productivity because once it’s automated, teams spend less time agonizing over whether or not their infrastructure is compliant, which gives them more time for innovation and feature development.

“While automation is an important part of DevOps, it’s by no means the only methodology also calls for cross-functional collaboration and the breaking down the silos and knowledge sharing across an org,” Ben Ezra said. “GitOps builds on these principles of leveraging automation and infrastructures as code to reduce configuration drift, and providing a single source of truth for an entire team or org.

By writing it down, all team members can contribute to the infrastructure code, which promotes shared responsibility across the entire software development lifecycle. Just as importantly, everyone is aware of these changes, so they can speak up if they see something you missed.”

“For example, at the New York Times, we’ve leveraged utilities from OPA to improve feedback and developer productivity within GitOps operational framework,” Ben Ezra said.

Principle #3: GitOps Apps Are Pulled Automatically

Software automation automatically pulls the declared state declaration from the source.

Take Flux, for example. Flux is a set of continuous and progressive delivery solutions for Kubernetes that are open and extensible, enabling GitOps and progressive delivery for developers and infrastructure teams.

“So Flux is a set of continuous delivery tools that are focused on security, velocity and reliability,” said Priyanka Pinky Ravi, a developer experience engineer with Weaveworks, in a keynote with other engineers from Cloud Native Computing Foundation graduated projects.

“And they are focused on automation. So the idea is that you have this setup Kubernetes controllers that you install onto your cluster, and they’re running on a reconciliation loop which is just an interval that you set. And every time that runs, the source controller will go in and pull from whatever source you said, such as a git repository, home or repo Image, Image registry, OCI registry. And the idea is that it pulls the manifests that it finds there and then actually applies them onto your cluster.”

Principal #4: GitOps Apps Are Continuously Reconciled

“Software agents continually observe the actual system state and attempt to apply the desired state.”

Of note are the different views that reflect the GitOps community. But with a set of principles, the community can build approaches that reflect the core focus of what GitOps is and what it is not; at least, that’s the concept.

Start reading the discussions on GitHub; there are still issues to clarify, even when explaining the meaning of pull.

And there’s more on the Twitter threads. Enjoy.

When
1. An operator pushes a button on GUI to change the running system state directly without updating the desired state
2. Failures leading to drift from the desired state

Now either
A) an agent changes the system back to correct state
Or B) system is in an unknowable state

— Alexis Richardson (@monadic) May 11, 2023

1) Any pipeline not stored in git (just setup in a ui for example)
2) A pipeline that doesn’t deal with desired state at all, (just running a build, testing an image, gathering metrics, doing a data transformation, updating a database etc)
3) Any pipeline operating without…

— Dan Garfield (@todaywasawesome) May 11, 2023

The post 4 Core Principles of GitOps appeared first on The New Stack.

Gaming developers face myriad challenges in their workflows due to a variety of unique requirements — including the different kinds of assets that are core components in games, file size, the different profiles of stakeholders in the process, friction with typical source-code management platforms and much more. These directly affect scalability, ease of use, collaboration with nontechnical stakeholders and compliance processes.

Source code management (SCM), which also serves as version control, has become a core piece to both asset management and developer productivity. Therefore, the one you choose can have a direct impact on your engineering velocity.

Below we’ll dive into each of the different options and the use cases they are optimized for.

How to Choose

Version control software is an essential tool for game developers to manage and track changes to their game’s source code as well as graphical assets throughout the development life cycle.

What should you look at when making such a choice?

1. Scalability
2. Simplicity and developer and user experience
3. Cost
4. Integration with existing tools and stacks

Scalability

The large size and complexity of game projects require version control software that can handle the storage and management of game assets and codebases efficiently and that can operate smoothly without slowing you down. This will directly affect how your engineering organization can grow and scale. You should select software that can support multiple developers, as well as the other critical stakeholders in the development process, and provide tools for tracking changes to game assets and resolving conflicts. Like all priorities when selecting software, how critical this is to your engineering processes is dependent on the size of your team, projects and the size of the assets used in them.

Simplicity and Developer/User Experience

Game development teams typically contain many members who are not highly technical, such as artists and designers who need to feel comfortable with the tooling the team is using. Depending on the technical skill level across your team, you should prioritize version-control tools that can be operated via simple-to-use interfaces and workflows, that don’t require deep technical skills or syntax and that are easy to learn and adopt.

Cost

Like all software, the cost of the tooling and software we choose can have a direct effect on our business’s bottom line. With the variety of version-control tools and price models available for them, you should be cognizant of this when making your decisions, with your technology needs and budget in mind.

Remember that since we are dealing with projects and assets of large size, storage and bandwidth costs come into play as well — be it your own operational expenses or as part of the subscription you are paying. Depending on your budget constraints, you may decide to go with a cheaper or completely free, tool and compromise on other criteria. When it comes to scale and growth, many of these tools are priced per user, and there are other factors that you will need to consider.

Integration with Existing Stacks

Software development typically involves the use of multiple development tools, such as issue trackers, project management software, and continuous integration/continuous deployment (CI/CD) tools that are critical to rolling out software the team is already familiar with and using on a day-to-day basis. Depending on the tools that are part of your workflow, you might need to prioritize version-control software that seamlessly integrates with as many of them as possible, so that you don’t have to invest too much time with integrating the version-control software into your stack but actually delivering value by writing code.

The Most Popular SCMs for Game Development

Git, Perforce, Plastic SCM and SVN are the most popular version control tools used by game developers (you can find more data on the different SCMs gaming developers use in this survey we recently conducted). Git is nearly the de facto version control tool in typical software engineering, as it is a very popular open source distributed version control tool among developers. The rest of the tools are actually more popular for game developers specifically and have unique advantages for them.

Perforce is the second most popular version control tool in the survey we conducted, after Git, and is a proprietary version control tool used by many game studios for centralized version control featuring high performance and scalability.

Plastic SCM, recently acquired by Unity, has undergone a facelift and is now known as Unity Version Control. It is a distributed version-control tool that focuses on game development workflows, particularly for Unity developers (a very popular game development engine and platform).

Apache Subversion (SVN) is a veteran SCM that is still in use today. It was Git before Git was popular, and it is an Apache Software Foundation open source centralized version-control tool that is among the tools used by game developers to overcome some of the unique challenges they face. It has features such as atomic commits and automatic conflict resolution.

Each of these tools has unique features and strengths. Deciding on a version-control tool will depend on the specific needs and preferences of the game development team.

Git: The De Facto Version-Control Tool for Software Developers

Git is the most popular version-control tool used by game developers (especially in small teams), mostly due to the fact that it is the standard de facto across the software-development industry. Git has associated SaaS services like Github, Gitlab and Bitbucket that provide additional features like hosting, code reviews and issue tracking, which played a big part in Git gaining widespread adoption. These services have different pricing plans, including a free tier, and they offer integrations with other development tools like IDEs, project management tools and build systems — and they even have some built into their platforms such as Github Actions and Gitlab Pipelines.

Git has the benefit of being a common tool across the industry, so most developers are familiar with it; however, using it can be challenging for nontechnical users, such as designers and artists, and even some technical users (we’ve spoken about Git and cognitive load in detail, too). Git requires a good understanding of its command-line interface and concepts like branching, merging and conflict resolution, and even when you do have this depth of understanding, you can find yourself getting into conflicts that are difficult to resolve. There are some clients who simplify the work with Git, but learning its syntax and concepts is difficult and may be a challenge to adopt, especially for non-technical staff.

Another consideration for gaming developers is that Git can encounter scalability issues for both repository size and file size, which can affect its performance for large game development projects. To manage large game assets, developers have to use Git-LFS, an extension that enables Git to handle large files more efficiently. However, using Git LFS adds complexity and is known to have issues, and it still has size and performance limitations.

The bottom line: Using Git is the simple choice if you are optimizing for your developers to feel at home — and you are fine with other stakeholders either having more difficulty adopting the tool or to not being a part of the code development process.  Although it is open source, note that self-hosting Git is not easy, and if you choose one of the SaaS-based offerings, it will come with its own cost implications. (In the words of Linus Torvalds himself, creator of Git: “Linux is free if your time is worthless.”) Keep that in mind when it comes to cost, and make sure you are aware that you will need to use a different tool for assets and large files.

Perforce: The Gaming Dev-Focused Version Control

Perforce (since the vendor expanded its portfolio, known specifically as HelixCore) is a popular version-control tool used by game developers (mostly in large companies) to manage source code and assets because it is known for its ability to handle large files and repos and support for distributed teams, all challenges well known to gaming developers. Like Git and SVN, Perforce is also a veteran technology, that’s been around since the early 1990s. However, unlike some of the other version control tools, it provides a scalable solution for managing large codebases and assets.

Perforce is a commercial product, with no open source or free option, and the licensing can be very expensive. That said, it does offer robust features and reliability, a balance that makes it popular primarily with AAA game vendors. Anyone who has ever attended Game Developers Conference (GDC), cannot help noticing the major presence of Perforce in the expo, they regard game developers as a target market and focus their marketing and engineering efforts there.

It is relatively simple to use and has extensive documentation, but it is still a technical-looking tool that is not intuitive for nontechnical personnel.

Perforce integrates well with a variety of developer tools used in the gaming industry, but integration with dev tools is not as broad as Git. However, it is definitely satisfactory for most scenarios. Some large customers are using tools they have built on top of Perforce to tailor it to their needs and to their users. Perforce does not have a true SaaS offering and requires you to deal with hosting the server (either on-premises or in the cloud). To use Perforce in a geographically distributed organization, even more significant IT efforts should be invested.

The bottom line: Use Perforce if you want to follow what many AAA game companies do — you get a robust version control tool that supports large assets and large projects. You will also be able to integrate with a variety of tools and have to compromise on user experience with non-techies. Note, however, that using Perforce does come with a price: It is the priciest version control offering, and it is not a turnkey SaaS offering. It requires you to dedicate the IT resources to support it.

SVN: Battle-Tested and Simple

Subversion (SVN) is another popular open source version-control tool used by game developers to manage source code and assets. It was the de facto version control for all developers before Git gained widespread adoption. SVN has been around since 2000 and has been widely adopted by many development teams, and it continues to be supported by the Apache Software Foundation.

SVN is known for its ease of use, with a simple command-line interface and user-friendly GUI clients, which some developers still prefer over more popular SCMs. SVN is also highly scalable, able to handle large codebases and large assets with ease, which makes it popular with game developers.

Like Git there are some companies that offer hosted and SaaS SVN; however, the service is not cloud native.  The cost of using SVN is generally low, as it is open source and free to use, but it comes with maintenance overhead, which is an operational cost factor that should be considered and should not be taken lightly (which is why many customers prefer to pay a provider and have it hosted). SVN integrates well with a variety of developer tools and services, but not as extensively as other contenders in this space. Some developers find that SVN’s branching and merging features are not as robust as other version control tools, and it may not be the best option for large teams and complex projects.

The bottom line: Use SVN if you’re tight on budget, want a simple but working solution and are willing to settle for a barebones SCM. It will support your large files and repos, but it may be less than ideal for complex workflows. Your developers will get most of the integrations they need, but they may have to compromise on user experience.

Unity Version Control (Previously Known as Plastic-SCM)

Plastic SCM was created in 2005 by Codice Software and has gained popularity among game development teams due to its focus on this community, providing an intuitive and user-friendly interface, making it easy for developers, as well as non-developers, to use. In 2021, Codice Software was acquired by Unity Technologies, which develops a game engine and development platform, and has since been adapted and optimized for Unity developers.

Plastic SCM was built with game devs in mind, so it is able to handle large codebases and assets, and Unity is now leveraging this value to benefit Unity developers in particular.

While there is a cost associated with using Unity Version Control, the pricing is flexible and depends on the size of the team and project, as well as the amount of storage required.  Plastic SCM used to integrate well with the common developer tools game devs use, but since being acquired by Unity, it’s been focused on integration within the Unity product suite. It remains to be seen how support for other toolsets and platforms will evolve and be maintained in the long run.

The bottom line: Unity Version Control is a great choice if you’re a Unity developer, and you find the all-in-one concept appealing (which could also play out well in terms of pricing). It does provide the scalability you need and a relatively simple interface for non-devs. However, if you have cross-platform development needs, or aren’t solely a Unity shop, this may be an important consideration going forward.

Diversion: Cloud Native SCM

Diversion is a new player in the version control space to help evolve an ecosystem that has not changed much over the past two decades. The world of SCMs for game devs as an industry is even further underserved than the general software industry. Diversion has entered the playing field with the ambitious goal of targeting these challenges and harnessing the power of modern, cloud native capabilities to take game development to its next phase of growth.

Diversion is described as a cloud native, SaaS version control which leverages the benefits of the cloud to deliver scalable, instantly available and simple to use version control. It is set to enable the management of all assets — code and art — in a single repo without limitations on size or geographical locations, and it is set to support artists in the same interface and platform.

The bottom line: Diversion is a new player (currently in public beta), up against longstanding incumbent and widely adopted SCMs. With a robust and wide-ranging feature roadmap, it is well-positioned to take on known challenges in the game development space, though not all of these features are currently available (or may still be in the works). With SaaS-based pricing, it is aligned with other hosted tools, and even has a free tier. Built with a cloud-first mindset, it is a contender to disrupt the version control ecosystem; however, it is yet to be seen if it will deliver on this promise.

Cloud Native Trends for Gaming Development

The world of engineering has undergone significant changes over the last two decades, and version controls and other toolsets need to up their game and embed similar practices for the gaming development world.

We can see new trends in cloud native version control development from companies like Diversion, new CI/CD platforms and practices with tools like Dagger and Buildkite, along with many other improvements to longstanding technologies. Some of these haven’t evolved at the pace of engineering either, with the most common tools being decades old and could benefit from a hard reset in some cases, or just leveling up their game to the modern cloud native world.

These are all practices and improvements that the gaming development world should be able to adopt more rapidly and embed in their toolsets and stacks, to support the unique challenges and growing popularity of gaming around the world.

The post 5 Version-Control Tools Game Developers Should Know About appeared first on The New Stack.

Responsible organizations already spend a lot of time and resources building secure pipelines, implementing best practices and testing applications for vulnerabilities. Despite all these efforts, there’s a hidden risk that is often overlooked in the supply chain story: off-pipeline changes.

What happens when software changes aren’t properly vetted and approved through the established security process? What if they introduce vulnerabilities that go undetected until it’s too late? What if a nefarious insider or attacker runs workloads in your infra?

I want to emphasize the importance of tracking and monitoring all deployments, especially those that bypass the standard security protocols. Let’s discuss some of the risks and consequences of unapproved deployments, and explore some best practices for keeping your system secure and compliant, no matter how fast you need to move.

How Secure Are Golden Pipelines/Golden Paths?

For today’s software organizations, security has never been more top of mind. On one side there is the growing threat of being hacked by malicious actors, which is discussed in CrowdStrike’s 2023 Global Threat Report. And on the other, there is a wave of cybersecurity regulation from the government to mitigate such cybersecurity vulnerabilities. Software organizations feel the heat from both sides as they work to improve their security posture in ways that will also achieve audit and compliance with new rules.

To meet these challenges we hear a lot about the software supply chain and the importance of securing it with DevSecOps approaches and golden paths to production. Since the Biden administration’s executive order in 2021, we’ve heard a lot about software bill of materials (SBOMs), and new products, services and vendors have sprung up to help us with them. But, do DevSecOps and supply chain approaches meet our cybersecurity challenges head on?

In regulated industries, governance in development is a key focus when it comes to securing the software delivery pipeline. Teams are often required to follow strict guidelines to ensure the security and integrity of their systems.

Many organizations do this with what’s often referred to as a “golden pipeline” that incorporates all of the necessary security checks like pull-request checks, SAST, DAST, software composition analysis and so on. A typical example of a golden pipeline looks something like this:

Most teams follow these golden pipelines to production to ensure that their software delivery process is secure and compliant. And there’s a lot of benefits to following this approach:

• It takes the burden off each team to figure out how to comply.
• It allows all the stakeholders to collaborate and reach a consensus.
• It makes audit and change management easier.

However, while DevSecOps processes like these have been effective in controlling and monitoring the delivery pipeline, they do not secure the entire software development life cycle. Bad actors can bypass these controls and introduce unapproved changes directly into production, putting the entire organization at risk. Let’s probe a bit further and find out where the vulnerabilities are.

The Limitations of DevSecOps Pipelines

DevSecOps pipelines and golden paths are put in place to ensure that changes made to a system follow a defined process and are authorized before deployment. This helps maintain system stability, ensure compliance and mitigate risks.

But pipeline controls have one obvious limitation when it comes to ensuring the security and compliance of an entire software system. They can only ensure security and compliance for changes that have gone through the pipeline. They do not account for bad actors who access production by going around the golden path.

There are several key security questions that cannot be answered in the pipeline:

• How do we discover workloads that haven’t gone through our pipeline?
• What happens if an internal developer has the keys to production?
• What happens if we are breached?
• What happens if our deployment process has silent failures?

Think of a golden pipeline as a river running into a lake. Monitoring what’s flowing in the river does not guarantee the quality of the water in the lake. You need to monitor the quality of the water in the lake too!

Closing the Security Loop with Runtime Monitoring

These security questions that can’t be answered in the pipeline can be answered in runtime. Monitoring production is the only way to be sure about what’s actually running in your environments and adds an extra layer of security on golden pipelines. It’s also quite straightforward:

• Record every approved deployment in the pipeline.
• Monitor (and record) what is running in production.
• Raise the alarm if any unexpected workloads are running.

By monitoring and recording the state of production, organizations can detect any unauthorized changes that have bypassed the change management process. This closes the loop on security, ensures continuous compliance in your environments, and greatly mitigates threats and risks.

The Benefits for Security Audits (and Auditors)

For many teams, particularly in regulated industries like finance and health care, software delivery must be managed according to a specific process to manage risks. And to ensure that the process is followed for each change, a change management process is added for verification before deployment. Traditionally, it is a gated process of manual controls and approvals.

Change management implemented as a gate has a lot of negatives:

• Increases lead time for changes
• Leads to larger and riskier releases
• Requires manual audits to ensure compliance

But taking a golden pipeline approach to the change management process builds compliance into every change. By recording the evidence for every test, security scan, pull request, etc., as they happen in the pipeline, proof that the delivery process is being followed and risk controls are being met can be gathered automatically. It’s more reliable, faster and removes the need for a manual gate.

This approach turns subjective processes into objective, high-efficacy risk management. It also has the added benefit of enabling automated, continuous auditing. You don’t have to spend time gathering the paperwork for an audit trail when all of the evidence has already been captured in your pipelines and then securely stored.

Toward Autonomous Security and Governance

Autonomous governance is an alternative approach to traditional change management processes. It brings compliance, risk, security and development teams together to define standards and automate controls, evidence gathering, and approval work. It leaves humans in the loop for governance while providing audit-ready, continuous compliance for deliveries.

With autonomous governance, organizations can identify and address non-compliant changes immediately instead of waiting for audit time. It’s faster, more effective and provides better risk mitigation than traditional change management processes.

Conclusion

DevSecOps approaches are effective in controlling security in the delivery pipeline, but they fall short in securing the entire software development life cycle. Monitoring production is a powerful addition that will help you to properly secure a DevSecOps approach. By keeping a close eye on production, organizations can detect any unauthorized changes and address them immediately.

With runtime monitoring and autonomous governance, organizations can bring compliance, risk, security and development teams together. In this way, the whole tech organization can work to define standards and automate controls, evidence gathering and approval work to ensure a much more secure DevOps environment.

The post Mitigate Risk Beyond the Supply Chain with Runtime Monitoring appeared first on The New Stack.

DORA metrics.

If you’re in the know, the acronym might make you shudder.

Organizations of all sizes are striving to implement DORA metrics into their processes, reporting, and development results, all to increase developer productivity.

While valuable, DORA metrics can be difficult to put in place, hard to track, and even harder to achieve top marks. Measuring these outcomes can only go so far if developers aren’t enabled with the right tools, processes, and expertise to meet them. If developers aren’t empowered to utilize DORA metrics to better their own performance, then they become just another mysterious set of numbers in a sea of unactionable data.

In this article, we highlight how to improve developer experience and enable better outcomes through continuous deployment, but there are many approaches to solving the DX problem at your company. What matters is that you identify that developer experience matters and make the effort to improve.

What are DORA metrics?

If you’re not in the know (and not shuddering at the thought), here’s a quick overview of DORA metrics.

DORA (DevOps Research and Assessment) metrics are the industry standard for measuring developer productivity, and provide important insights into how companies use their development teams. These metrics allow developers and technology leaders to understand where they should focus their efforts in order to increase efficiency. DORA metrics are used to assess:

• Deployment frequency – the frequency of successful software releases to production
• Change failure rate – the rate at which code changes cause breakages or failures
• Mean time to recovery – the time between deployment or system failure and recovery
• Lead time for changes – the length of time between a code change commit and deployability

They provide a clear, objective way to measure and track developer performance, and are an invaluable tool for companies and developers alike. By taking advantage of these metrics, companies can ensure that their development teams remain productive and efficient, while developers can use them to get an accurate assessment of their performance.

How Does DORA Impact Developer Experience?

DORA metrics can help developers to understand the overall health of their code and can be used to identify specific areas that need improvement. Using these metrics, developers can identify problems early in the development stage and make changes accordingly, leading to improved performance and cost savings in the long run. DORA metrics provide real-time feedback which can help developers to stay on top of changes and respond quickly to evolving requirements.

Overall, DORA metrics are an invaluable tool to maximize the success of projects. By leveraging these metrics, developers can gain a better understanding of the impact of their code changes, identify areas that need attention, and quickly respond to changing requirements. This can help developers achieve higher levels of success and improve their development experience overall.

Why Does Developer Experience Matter?

​​Developer experience (DX) matters because it can make or break the success of a developer’s project. DX is about making sure that developers have all the tools, support and guidance they need to complete their tasks in an efficient and satisfying way. It’s not just about providing developer-friendly tools and documentation: great DX also includes creating a developer-friendly environment, offering helpful advice and feedback, and making sure developers have the resources to succeed.

By focusing on DX, companies can ensure that their developer projects are successful, efficient, and enjoyable. This will not only benefit the developer but also the company as a whole by improving developer retention rates and fostering meaningful collaboration between teams.

Layering DORA metrics on top of a shaky foundation with no consideration for tools, communication, processes, and feedback can create negativity among the development team.

So What Is Continuous Deployment?

Switching gears here from establishing the need for DORA metrics into possible solutions, let’s focus on continuous deployment.

Continuous deployment is a process of continuous software delivery where every code change goes through the same automated process, allowing new features to be released in an agile manner. It reduces deployment time and makes it easier for teams to iterate and quickly deploy changes without any manual intervention.

This allows teams to rapidly deliver better customer experiences and shorter release cycles. Teams are able to easily deploy changes when necessary and can quickly respond to any customer needs or requests that may arise. This helps ensure a positive user experience and continuous improvement of the product.

The Cloud Native Computing Foundation (CNCF) hosts critical components of cloud native software stacks, including Kubernetes and Prometheus. CNCF serves as the neutral home for collaboration and brings together the industry’s top developers, end users and vendors.

Learn More

The latest from CNCF

How Does Continuous Deployment Impact Developer Experience?

Continuous deployment is a great tool for developers as it can give them more control over the development process and help them stay organized. It also allows developers to test their code quickly,  push changes out faster, and enables developers to get feedback on their work quicker than with traditional methods.

By using continuous deployment, developers can increase their productivity and make sure they are creating quality products for their customers. This makes it easier for them to stay motivated, which in turn leads to a better developer experience.

All of these factors can have a positive impact on developer experience and help developers stay focused on the quality of their work, not just on DevOps tasks.

How Does Continuous Deployment Solve the DORA Problem?

Let’s break down the DORA metrics, what you or your developer team might be saying is an issue, and how continuous deployment solves them and impacts your DORA metrics.

• Deployment frequency –
  • The Development Team Might Say: “There’s so much red tape to push code live, that I spend more time waiting than writing code.”
  • Continuous Deployment DORA Impact: Deployment frequency goes up; it’s always available, and devs can push code whenever they’re ready, not when a manual gatekeeper allows them to.
• Change failure rate –
  • The Development Team Might Say: “I need to know if the code I’m committing will cause a breakage, causing me to redo work that I’ve already done. Waiting for the Ops team to test and approve is a barrier for me to continue developing.”
  • Continuous Deployment DORA Impact: Rate of failure goes down with continuous deployment because you can test automatically and implement automated canary and blue/green deployments, ensuring you’re not releasing breaking changes.
• Mean time to recovery –
  • The Development Team Might Say: “When I receive my code back hours after I commit because it causes something to break, it takes me longer to assess the problem because I’ve already moved on from that commit.”
  • Continuous Deployment DORA Impact: With good continuous deployment tools and practices also comes easy, automated roll-backs, leading to faster recovery and immediate fixes if a breaking change does occur.
• Lead time for changes –
  • The Development Team Might Say: “I spend a lot of time checking and rechecking my code because I know if it goes out, despite all of our testing, it has the potential to cause an outage. I’m afraid to commit my code because the recovery process is long and painful.”
  • Continuous Deployment DORA Impact: Enabled developers create better code, faster. With continuous deployment in place, teams can commit with confidence knowing that their code will be deployable. Even if it isn’t, then it won’t go to production and won’t break anything!

As we said in the intro, DORA metrics can only get your organization so far. You need to couple them with good developer experience through tools, processes, and enablement.

A development team that is properly enabled through continuous deployment benefits your entire organization. When devs feel connected to their work, invested in success, and can work efficiently and independently, they’re often happier, code better, and release features more quickly, helping bring more value faster.

The post Level up the Dev Experience with DORA Metrics and Continuous Deployment appeared first on The New Stack.

Just as everyone was heading out to the New Year’s holidays last year, CircleCI CTO Rob Zuber got a surprise of a most unwelcome sort. A customer alerted CircleCI to suspicious GitHub OAuth activity. Although the scope of the attack appeared limited, there was still no telling if other customers of the DevOps-friendly continuous integration and continuous delivery platform were impacted.

This notification kicked off a deeper review by CircleCI’s security team with GitHub, and they rotated all GitHub OAuth tokens on behalf of their customers. On January 4, the company also made the difficult but necessary decision to alert customers of this “security instance,” asking them to immediately rotate any and all stored secrets and review internal logs for any unauthorized access.

In this latest episode of The New Stack Makers podcast, we discuss with Zuber the attack and how CircleCI responded. We also talk about what other companies should do to avoid the same situation, and what to do should it happen again.

“I think it’s a tough decision. It shouldn’t be a tough decision. But it is a tough decision, I think when something like this happens, it feels like you’re putting yourself at risk,” Zuber said, on making the decision on alerting all customers about an attack on a single client. “But the reality of the situation is for an event of the scope, and size and impact on our customers, there weren’t a lot of choices. What the best thing that we could do for our customers was what drove all of our decision making.”

After the first disclosure, CircleCI followed up with customers offering a more detailed examination of what took place. This transparency was important for the company, and CircleCI openly published details on every public platform it could.

“It was important to us to make sure that every customer could check their own exposure,” Zuber said. “There was not really a model for us to figure out every single customer that was impacted, only those that could be, which is the full scope of our customer base, effectively, then informing those customers [on] what impact would look like to the best of our understanding.”

It turned out that the attack was just on the one customer. Nonetheless, CircleCI did make some changes in its tooling, and in scaling down the scope in which access permissions are granted. In general, the company has started moving away from “a model where you would have secrets with us and into one where you’re doing a secure token exchange on the fly as part of the build,” Zuber said.

And this is where the industry needs to head, Zuber said. “We can always enhance how we’re managing things ourselves in terms of our delivery, and in terms of the system that we’re running. But if the exposure inside that system is the short-lived tightly-scoped tokens, then the overall sort of potential [for a breach] is massively reduced in the future,” he said.

“The place that we’re really interested in is not just changing our product, but helping the industry move forward.”

The post CircleCI CTO on How to Quickly Recover from a Malicious Hack appeared first on The New Stack.

The Slim Developer Platform aims takes the pain out of vulnerability remediation and management for container-based applications. The platform can reduce vulnerability counts by 80% (on average) and equips developers and security professionals with tools to understand which vulnerabilities matter and which don’t. Using proprietary “container hardening” algorithms based on the ultra-popular SlimToolkit open source project, Slim removes unnecessary libraries, packages and binaries, thus minimizing a container’s attack surface and the potential for zero-day attacks.

Differentiator

Top differentiators of Slim.AI’s platform include the following:

1. Slim.AI provides proactive vulnerability remediation. While most software supply-chain companies are focused on generating awareness of existing vulnerabilities (through vulnerability scanning or Software Bills of Material a.k.a “SBOMs”), Slim.AI reduces production applications to a minimal footprint proactively, removing potential future threats.
2. Slim.AI is focused on automation for any technology stack. Previous approaches to slimming containers can result in manual effort for developers or ask developers to change their base image, distribution, or package ecosystem. Slim’s goal is to let developers work however they want and to provide trustworthy automations that run in CI/CD with every build. This approach decreases the friction between developer teams and security/compliance teams — a win-win.
3. The Slim Developer Platform is built on SlimToolkit open source software (16K GitHub stars and growing), which many organizations have already embraced as a valuable tool for modernizing their cloud native workflows. Slim.AI makes using SlimToolkit easier, faster and more scalable for teams of developers worldwide.

Automated vulnerability remediation is gathering steam. Several startups — such as RapidFort, Chainguard and EndorLabs — are focused on the problem, though all have different approaches. Additionally, there are several existing methods for managing container vulnerabilities, including:

1. Alternative base images: Alpine Linux, Distroless and Scratch images ask developers to start with a minimal image and add the tools, packages and libraries they need to it. For some developers, these approaches are challenging due to low-level differences in the distributions or lack of understanding as to how these techniques work.
2. Vulnerability scanners and SBOMs: While a critical part of a secure posture, these technologies are point-in-time and reactive solutions to security. They can create friction for development teams and don’t address other aspects of attack surface outside of vulnerabilities and package information.
3. Policy engines: These rules-based engines can prevent risky containers or configurations from reaching production and are necessary to ensure compliance. However, they tend to be a “red light” approach to security and can have a negative impact on developer velocity.

Slim.AI is focused on containers as the atomic unit of a secure cloud native posture and is the only company offering a proven, trusted method for automatically hardening containers en route to production. Being a SaaS service lets Slim.AI connect with multiple cloud providers (Amazon Web Services, GCR, Azure, etc.), but also facilitates team collaboration, sharing and reuse of important artifacts for delivery and security.

Problem Space

Large, unoptimized containers can be rife with vulnerabilities and additional attack surface (see Slim.AI’s annual Public Container Report for more information); yet, to date, hardening containers is a highly specialized and labor-intensive job.

Benefits of Slim.AI

Slim.AI seeks to be a communication platform between container producers (software companies shipping products to customers in the form of containers) and container consumers (their customers). By reducing the attack surface of a container (i.e., removing shells and package managers), the exploitability of a given vulnerability is greatly reduced.

Company

In 2015, the Docker community held a Global Hack Day in Seattle. Kyle Quest’s concept for “DockerSlim,” which he described as “a magic diet pill for your containers,” won first place in the local event and second place in the global “plumbing” category that year.

That’s how the seeds were sown for an open source community that now supports SlimToolkit. Around 2019, the project had achieved so much momentum that users were regularly asking for extended features and additional functionality. That spurred Kyle and John Amaral to put together a business plan. Quest and John Amaral launched Slim.AI in 2020 (as founding CTO and founding CEO, respectively) on the premise that true software security comes from within. The company’s vision is to empower developers to employ container best practices to deliver not only more efficient and performant software but more secure software, as well.

Stack

The Slim platform can analyze and harden any OCI-compliant container image, regardless of its base image, package ecosystem or build origin. While the SlimToolkit open source software requires the Docker daemon, Slim’s Automated Container Hardening doesn’t and can be used with any runtime, including ContainerD/Kubernetes.

Images should be hosted in one of the many cloud registries supported by Slim (e.g., Docker Hub, AWS Elastic Container Registry, Google Container Registry, Microsoft/Azure, RedHat Quay, GitHub Container Registry and others). Additionally, Slim supports several CI/CD system integrations including GitHub Actions, CircleCI Orbs, GitLab and Jenkins.

While Slim supports both Linux/AMD- and ARM-based image architectures, cross-architecture builds are currently not supported. Additionally, Slim’s core hardening capability requires a secured connection to the Slim platform, though air-gapped and on-premises solutions are on the near-term roadmap.

Partnerships/Customers

Numerous Slim.AI design partners have testified to the impact of the Slim.AI platform; here are a few who have documented their experiences and results:

• BigID: BigID automates container security with Slim.AI to reduce vulnerabilities and maximize security posture. Learn more about BigID and Slim.AI on this episode of TFiR.
• PaymentWorks: PaymentWorks used Slim.AI to eliminate 80% of container vulnerabilities with no additional developer overhead. Read the PaymentWorks case study.
• Jit: Jit achieved a step change in DevX with minimal integration effort, reducing container size by 90% and cutting bootstrap time in half. Read the Jit case study.
• Security Risk Advisors: SRA sought to deploy modern processes like containerization, slimming, SBOMs (software bills of materials) and vulnerability management without having to largely expand the DevOps team, and it found the ideal solution in Slim.AI.

Pricing

The Slim.AI platform is currently in beta and available for free to developers. Developers can log in to the Slim.AI platform to analyze their containers, get vulnerability reports from multiple scanners and automatically harden their container images for production.

Additionally, Slim.AI has been adding functionality for teams and is accepting a limited number of organizations into its Design Partner Program.

For more information, contact ian.riopel@slim.ai.

The post Tech Backgrounder: Slim.AI Makes Container Hardening Easier appeared first on The New Stack.

Chaos engineering is at the stage where disaster recovery was a few years ago in many ways. What typically happened back in the day before disaster recovery became widely accepted by the IT community is that a prolonged data center or server outage typically occurred leading to a catastrophic failure when the disaster recovery system did not function as it should — or even worse — did not even exist. Even more tragically, the IT team often realized after the incident that there had been obvious signs that an outage was imminent but they had failed to heed those signs because they did not know where to look. This is where chaos engineering comes into play.

Chaos engineering has emerged as an increasingly essential process to maintain reliability for applications — or in not only cloud native but any IT environment. Unlike pre-production testing, chaos engineering involves determining when and how software might break in production by testing it in a non-production scenario.

In this way, chaos engineering becomes an essential way to prevent outages long before they happen. It is defined as an overlap between reliability testing and experimenting with code and applications across a continuous integration/continuous delivery (CI/CD) pipeline and for maintenance and management tasks once deployed. Chaos engineering is achieved by obtaining metrics and data about how an application might fail when certain errors are induced through experiments. The end result is continuous resilience across CI/CD and operations.

“Chaos engineering gives us a way to practice our professional skills, especially with a focus on improving time to recovery,” Adrian Hornsby, principal system dev engineer for AWS said during his keynote while noting that the average per hour for infrastructure downtime is $100,000, according to IDC stats. “It helps us build a culture of resilience.”

How chaos engineering is used to mitigate downtime threats of money lost during downtime or worse yet, unrecoverable failure, its adoption path and tools involved were some of the main themes discussed during Harness’ annual users’ conference, Chaos Carnival, in March.

Hard Facts

With the rise of cloud native applications reaching a total market size of $17 billion by 2028, chaos engineering and proactive resilience testing tools and services will be leveraged to ensure enterprises achieve maximum system availability and developer productivity, Prithvi Raj, technical community manager for Harness and a community lead for CNCF project LitmusChaos, estimates. All told, nine out of the top 15 banks in the U.S. are pursuing chaos engineering and six out of the top 10 biggest retail companies in the U.S. have already brought in some form of chaos engineering as a practice.

“Potentially, chaos has seen the highest demand from banking, fintech, e-commerce and telecommunication with other sectors picking up. The cloud native chaos engineering space has grown over ten-fold in the last year alone and we are experiencing almost 60,000 installations per month with an average of at least seven to eight-experiment runs per installation per month for just the open source tools out there,” Raj said. “While stats show that demographically the U.S., Canada, India, China and Mexico have the most chaos experiment runs per month, growth in Latin America and Europe has been phenomenal with Brazil, France and Germany picking up.”

The key takeaways for chaos engineering today and the Chaos Carnival conference that Raj communicated include:

• The true potential of chaos engineering can “be unleashed by embracing the practice and removing negativity.”
• Chaos engineering integrations with CI/CD tools are vital today to adhere to cloud native dependencies and the continuous system changes from the rapid release frequency of developers.
• The struggles of chaos engineering are based on “organizational structure and truth and the road ahead requires perseverance.”
• Documenting chaos engineering is important just like any other framework to avoid the trial-and-error way of functioning.
• Security chaos engineering has enabled achieving automated cloud security and “is the need of the hour.”
• The road to building chaos engineering ahead is community collaboration and “working with the CNCF to help open source projects grow.”

Observability Link

One of the main results of chaos engineering is continuous resiliency. To wit, it is analogous to monitoring and observability to which chaos engineering is related.

In order to achieve observability, astute monitoring is required. It is the act of proper monitoring of systems, software operations, CI/CD pipelines, etc. that leads to the state of observability required to make decisions based on the ability to process the data.

What's your favorite way to build resilience in your software and teams? We asked some attendees at Chaos Carnival 2023:

I appreciate the honesty!

The answers improve resilience across people, processes, and technology!#chaosengineering #chaoscarnival pic.twitter.com/HfbXnlNLTv

— Chaos Carnival 2023 (@chaoscarnivalio) March 16, 2023

Chaos engineering, when done properly, requires observability. Problems and issues that can cause outages and the greater performance can be detected well ahead of time as bugs, poor performance, security vulnerabilities, etc. become manifest during a proper chaos engineering experiment. Once these bugs and kinks that can potentially lead to outages if left unheeded are detected and resolved, true continued resiliency in DevOps can be achieved.

In the event of a failure, the SRE or operations person seeking the source of error is often overloaded with information. “You’ve got to be able to reduce everything to actionable insights rather than just having every dashboard read and everything log browser just scrolling up the screen faster than you can read. And your observability system not only needs to stay up when everything’s down — so you probably want it on some different infrastructure — but it also needs to cope with floods of alerts without failing,” Adrian Cockcroft, a technical advisor  And then to really do that to show that that the way you’re set-up, you need to run these regular chaos engineering experiments.”

LitmusChaos Test

Chaos engineering requires experiments and tests. Among the tools available to that, Harness’s LitmusChaos open source project and enterprise versions were discussed during the conference. ChaosNative Litmus Enterprise has helped DevOps and site reliability engineers (SREs) to adopt chaos engineering tools that are self-managed, while the cloud service, ChaosNative Litmus Cloud, offers a hosted LitmusChaos control plane. On offer are over 50 chaos experiments that LitmusChaos has already developed for you,” Uma Mukkara, head of chaos engineering for Harness, said during his conference keynote. They cover a range of Kubernetes resources cloud platforms such as AWS, GCP and Azure, and applications such as Cassandra Kafka. “When it comes to letting us use cases, you can start continuous chaos testing with litmus or start practicing game days or start using chaos engineering and your performance engineering testbed or you can start integrating observability and chaos together, Mukkara said.

The post There Is No Resilience without Chaos appeared first on The New Stack.

Should Site Reliability Engineering alerts be “shifted left” into the Continuous Integration stage of software deployment, that is before the software is even deployed?

A recent Usenix opinion piece, “CI is Alerting,” written by Titus Winters, Principal Software Engineer at Google, explains how this potential practice could be useful.

As Winters points out, CI systems are systems for automating the build-and-test routine: build the code, and run the tests as often as is reasonable. Adding site reliability engineering alerts to CI should be possible, given that CI alerts should be treated the same way and tested with the same criteria as production alerts. That means CI shouldn’t have 100% passing rates and an Error’s Budget should be added. Brittleness is a leading cause for non-actionable alerts and flaky tests. This can be solved by adding more high-level expressive infrastructure.

Although CI and alerting are guided by different groups, the article makes the argument that they serve the same purpose and even that they use the same data at times. CI on large-scale integration tests are the equivalent to canary deployments and when using high-fidelity test data, reporting large-scale integration test failures in staging are basically the same failures seen in production alerts.

There is a purpose to the parallels. The purpose is that what works for CI can work for alerting and what doesn’t work for CI might not work for alerting and vice versa. This paves the way for the concept of brittleness being a problem.

“Given the higher stakes involved, it’s perhaps unsurprising that SRE has put a lot of thought into best practices surrounding monitoring and alerting, while CI has traditionally been viewed as a bit more of a luxury feature,” Winters writes. “For the next few years, the task will be to see where existing SRE practice can be reconceptualized in a CI context to help explain testing and CI.”

How Alerting Is Like CI

Here’s a production alert:

Engineer 1: “We got a 2% bump in retries in the past hour, which put us over the alerting threshold for retries per day.”

Engineer 2: “Is the system suffering as a result? Are users noticing increased latency or increased failed requests?”

Engineer 1: “No.”

Engineer 2: “Then … ignore the alert I guess. Or update the failure threshold.”

The alerting threshold is brittle but it didn’t come out of thin air.  Even if there was no fundamental truth to the specific alert, it’s correlated to what actually matters — degradation in service.

Here’s a unit test failure:

Engineer 1: “We got a test failure from our CI system. The image renderer test is failing after someone upgraded the JPEG compressor library.”

Engineer 2: “How is the test failing?”

Engineer 1: “Looks like we get a different sequence of bytes out of the compressor than we did previously.”

Engineer 2: “Do they render the same?”

Engineer 1: “Basically.”

Engineer 2: “Then … ignore the alert I guess. Or update the test.”

Similarly to the alert, the test failed on criteria that didn’t fully apply. The specific sequence of bytes doesn’t matter as long as the bitmap produced by decoding it as a JPEG is well-encoded and visually similar.

This happens when there isn’t enough high-level expressive infrastructure to easily assert the condition that actually matters. The next best thing is to test or monitor for the easy-to-express-but-brittle condition.

The Trouble with Brittleness

When an end-to-end probe isn’t revealed but collecting aggregate statistics is available, teams are likely to write threshold alerts on arbitrary statistics. In lieu of a high-level way to say, “Fail the test if the decoded image isn’t roughly the same as the decoded image,” teams will test byte streams. Such brittleness reduces the value of testing and alerting by triggering false positives but also serves as a clear indication of where it may be valuable to invest in higher-level design.

Just because brittle isn’t best doesn’t mean brittle is bad. These tests and alerts still point to something that might be actionable. Data surrounding the alert will lead to more clues about the importance of the alert. This is why Winters explains that flaky tests are more negatively impactful than non-actionable alerts. There’s usually fewer data in the testing environment to show whether or not the test failed because of a software-related or test-related issue.

What Is the Pathway Forward?

Treat every alert with the priority it deserves rather than always being on high alert. Consider adding the flexibility of an Errors Budget to CI rather than only having an Errors Budget in alerting and focusing on absolutes with CI. Winters views that as a narrow perspective and recommends refining objectives and adding in an Error Budget for CI because 100% passing rate on CI is just like 100% uptime: awfully expensive.

Some other lessons learned:

Treating every alert as an equal cause for alarm isn’t generally the right approach. This is one of those “alarm snooze” situations. The alarm matters but if it’s not incredibly impactful it’s ok to keep on moving. But that also doesn’t mean the alarm should get thrown out the window because tomorrow is a new day.

Reconsider policies where, if not all CI results are green, no commits can be made. Don’t throw out the alarm — if CI is reporting an issue, investigate. If the root cause is well-understood and won’t affect production then blocking commits quite possibly isn’t the best pathway forward and could be problematic in the long run.

This is a novel idea and Winters says he’s “still figuring out how to fully draw parallels.” For the next few years, “the task will be to see where existing SRE practice can be reconceptualized in a CI context to help explain testing and CI.” He looks for best practices in testing to clarify goals and policies on monitoring and alerting.

The post Usenix: Continuous Integration Is Just SRE Alerting ‘Shifted Left’ appeared first on The New Stack.

When software developers are building their microservices for use inside Docker containers, it saves a great deal of time and effort for them to also test for various dependencies in parallel instead of starting from scratch after the app is done. As those dependency dots are connected, they often require changes to the app, which means doubling back and re-doing the code. And nobody likes re-doing anything.

This is where something new called Testcontainers comes to the proverbial rescue. Testcontainers is a library originally written in Java that helps developers run module-specific Docker containers while the app is being built in order to simplify integration testing. These Docker containers are lightweight, and once the tests are finished, the containers are destroyed, and developers can move on to the next project.

Modules Program

Software company AtomicJar, a pioneer in this sector, on April 5 launched its Testcontainers Official Modules program with the backing of several major vendors. Redpanda, StreamNative, Neo4j, Cockroach Labs, LocalStack, Oracle and Yugabyte were among the first to declare support for Testcontainers.

The modules catalog features more than 50 modules supporting a list of often-used technologies and provides certification, support, and maintenance of Testcontainers modules for the development community. Each community partner is committed to supporting the program as the preferred way of testing their work and to developing with other partners locally.

“Testcontainers allow developers to test and develop their code against the real dependencies they will use when the app goes live for use,” Eli Aleyner, co-founder of AtomicJar, told The New Stack. “For example, a developer could write a test that is to be executed with a real instance of Kafka, MySQL or any other technology. When the test is complete, it will tear down any dependencies. This allows developers to create self-contained, repeatable and idempotent tests that can be run either locally or in the continuous integration process (CI).”

In the background, Testcontainers utilizes its own containers to spin up the dependencies, Aleyner said. “So when a developer uses Testcontainers to say: ‘I want an instance of Kafka,’ before the test runs, the Testcontainers library will fetch the Kafka container, start it locally, handle all the port mapping and other details automatically.”

The larger impact of this approach is that it enables organizations to give developers more control and allow them to get more confidence in the software they write before checking in their code, Aleyner said.

“Previously, the only place developers used to discover integration issues was during the CI process. With Testcontainers, developers are able to shift this to the left, find issues faster and iterate quicker,” Aleyner said.

There is no substitute for speed in agile software development, and tools like this one help developers stomp down on the accelerator.

Started in Java

The Testcontainers project started in 2015 in Java and has grown to include hundreds of thousands of instances of Postgres, Elastic, MySQL, and other enterprise components, Aleyner said. Testcontainers has since evolved beyond the Java ecosystem libraries into .Net, Go, Node.js, Python, Rust and Haskell as those communities begin to realize the value of quicker iteration enabled through this library.

Since its inception, Testcontainers library has been implemented in seven languages, and it has also been embraced by the development community from large to small companies, Aleyner said. DoorDash, Uber, Spotify, Netflix, Capital One, and several others have talked publicly about using Testcontainers to simplify their Testing setup, he said.

“In aggregate, we are currently tracking around 6 million Docker image pulls for Testcontainers a month,” Aleyner said. “We have seen the Testcontainers library being downloaded 100 million times in January of this year; we crossed 50 million downloads in May of last year, so the technology is getting a lot of traction.”

The post How Testcontainers Is Demonstrating Value as a Key CI Tool appeared first on The New Stack.

DevOps is full of buzzwords, jargon and abbreviations. DevOps itself has only been around for a little more than a decade, so some of these concepts are relatively new. However, some are quite old, and their definitions and uses have changed over time. Today, I’ll explore the history of Infrastructure as Code. It may seem new, but it’s both older and more historically complex than you might realize.

When someone says, “Infrastructure-as-Code,” your mind probably jumps to tools like Chef or Ansible or Terraform, but its origins are much older than that. It has its roots in configuration management, and most configuration management tools use some sort of Infrastructure-as-Code approach. It turns out that we’ve had trouble managing and configuring fleets of machines since we first decided to let them talk to one another, whether they’re virtual machines or containers on a cloud provider, or physical machines in a lab.

If you dig in and go all the way back to the dawn of modern computing, configuration management tools have been around since the 1970s. Unix “make” was released in 1976, allowing a rudimentary form of configuration management, and PXE boot was introduced in 1981, bringing an early form of configuration management to whole machines.

Both tools are still widely used today, but they don’t jump to mind when most people think of configuration management or Infrastructure as Code. You could easily argue that neither of these tools is true configuration management, but the fact remains that we used these tools to automate configuration to the best of our abilities before we had more sophisticated tooling to do it for us.

Dell Technologies (NYSE:DELL) is a unique family of businesses that helps organizations and individuals build their digital future and transform how they work, live and play. The company provides customers with the industry’s broadest and most innovative technology and services portfolio spanning from edge to core to cloud.

Learn More

The latest from Dell Technologies

One of the oldest open source examples of the modern definition of configuration management is CFEngine, initially released in 1993. It was built by Mark Burgess while he was pursuing a postdoctorate in theoretical physics. Mark had the task of maintaining a series of Unix workstations, all running different versions of Unix and having different problems, which required a lot of time-consuming manual scripting and one-on-one user support.

Abstracting away the differences between those platforms behind a domain-specific language (DSL) allowed him to significantly reduce the workload involved in maintaining them, and CFEngine1 was born. This style of configuration management served us well for a long time, and CFEngine is generally regarded as the earliest fully formed origin of today’s tools.

Figure 1

So why does Infrastructure as Code seem so new if configuration management is so old? Did the complexity that made configuration management useful not exist outside of academia and enterprises, or what? No, but like many advancements in technology, a need to handle greater and greater complexity at scale has driven the evolution of Infrastructure as Code.

Prior to the ubiquity of cloud computing, provisioning computing resources meant acquiring new physical infrastructure. Utility computing, or the “pay for what you use” model, wasn’t widely available yet, so scaling to infinity wasn’t nearly as easy as it is today. But the threshold for what constitutes “at scale” changes and grows over time as technologies evolve, and in 2006, AWS released the first version of EC2.

Rapidly, scalability became everyone’s problem. EC2 made it easy for anyone to take up the exact amount of computing resources they needed, but tools for managing that infrastructure hadn’t kept up with the rapidly growing complexity of applications built for this environment.

Manually provisioning and managing instructions for potentially hundreds of different environments is slow and unreliable, so this led to the introduction of a new class of tools for configuring and managing your infrastructure.

In tandem, Puppet and Chef came to light. Puppet was released in 2005. It uses its own declarative domain-specific language to define the resources and state for a system. Puppet accomplished a lot of the same goals that CFEngine did, although in a different language, but drastically reduced the learning curve.

In 2009, we got Chef. Unlike Puppet, Chef was built around a Ruby DSL. While this means that you have access to a language that’s close to full-fledged Ruby, it could be less friendly for folks who come from an IT Ops background rather than a programming background. For a lot of users, the difference between these two tools comes down to whether or not you are more comfortable writing configuration files or something that looks more like a program.

In 2012, Ansible was released. Like CFEngine and Puppet, it uses a declarative, domain-specific language. However, unlike CFEngine, Puppet, and Chef, Ansible is agentless, meaning there is no piece of Ansible installed or running on the machine it controls. Instead, it works via a temporary SSH connection, which is pretty cool.

Figure 2

These tools were each revolutionary in their own ways, and they dominated the market for years. But we tend to think of them mostly as configuration management tools, even though they do some of the same jobs as modern Infrastructure-as-Code tools and can provision infrastructure for us in a code-like way. So where IS the line, and how blurry is it?

Initially, I wanted to say that it’s the difference between managing resources and state for an application rather than for an entire machine, but that isn’t entirely clear. Tools for configuring machines and operating systems rather than applications do exist in classes of their own under the umbrella of “configuration management.”

So instead, my opinion is this: Configuration management exists on its own with these aforementioned tools, and also exists as a part of the greater concept of Infrastructure as Code.

Maybe it’s more accurate to say that Infrastructure as Code is the natural evolution of configuration management. Therefore, how did this inevitability happen?

The growing popularity of container technologies and Docker’s introduction in 2013 caused complexity to increase radically once again, allowing for much more sophisticated solutions and architectures but causing new pain points with respect to scalability.

This is also right around when the concept of developing applications as a collection of microservices rather than monoliths really began to take off, a design paradigm that Docker is well-suited to handle. Suddenly, we need smarter tools built for this type of work.

In 2014, the same year Kubernetes was introduced, Terraform also launched. Like most of its predecessors, it provides broad infrastructure configuration management with a domain-specific language; but unlike its predecessors, it was designed primarily for the management IT operational workflows, which are now more diverse and complicated than ever. Terraform has become wildly popular and continues as a result of the growing community and expansion into the cloud.

We’ve come a very long way so far, and each of these tools has strengths (usually reflected by the time and circumstances in which they were built), but IT Ops teams and site reliability engineers are still left with an outsized responsibility for their infrastructure.

Meanwhile, software engineers are given tools that use DSLs they don’t know, and increasingly face situations that would benefit from the expressiveness of a programming language without a real way to handle that. In fact, outcrops like Pulumi start approaching tools from a programming language capability rather than DSL.

Each step along this tooling timeline has gotten us closer to the DevOps ideal. I’ve skipped over some tools here and there for simplicity’s sake, such as Pulumi, Nix or SaltStack, but suffice it to say that the most detailed timeline of the evolution from make files to full-fledged IaC is much longer than this long blog post.

Infrastructure as Code feels shiny and new, but like most things in computing, it’s actually quite old — we’re just coming at the same problems from different angles to work around the circumstances of the day and build abstraction layers for the ones we’ve already solved.

The post A Brief DevOps History: The Roots of Infrastructure as Code appeared first on The New Stack.

Development teams want to build applications quickly. But that often puts them at odds with the need for testing. Developers might code up to the last minute, leaving almost no time to find and fix vulnerabilities before deadlines.

To streamline the development process and improve velocity, “shift left” security helps developers find and remediate vulnerabilities earlier in the development process. This is a pivotal part of supporting the DevOps methodology.

As cloud computing empowers the adoption of DevSecOps, those teams also get a centralized platform for testing and deployment. But for DevOps teams to embrace the cloud, security has to be at the forefront of their considerations. For developers, that means making security a part of the continuous integration/continuous delivery (CI/CD) pipeline that forms the cornerstone of DevOps practices.

CrowdStrike has redefined modern cybersecurity with our advanced cloud-native platforms for protecting endpoints and cloud workloads, identity and data. CrowdStrike’s adversary-focused approach to CNAPP provides agent-based and agentless solutions delivered from the CrowdStrike Falcon® platform.

Learn More

The latest from CrowdStrike

The New Way to Secure Applications Better

The CI/CD pipeline is vital to supporting DevOps through the automation of building, testing and deploying applications. It is not enough to just scan applications after they are live. A shift-left approach to security should start the same second that DevOps teams begin developing the application and provisioning infrastructure. By using APIs, developers can integrate security into their toolsets and enable security teams to find problems early.

Speedy delivery of applications is not the enemy of security, though it can seem that way. Security is meant to be an enabler, an elixir that helps organizations use technology to reach their business goals. Making that a reality, however, requires making it a foundational part of the development process.

In research from CrowdStrike and Enterprise Strategy Group (ESG), 41% of respondents said that automating controls and processes via integration with the software development life cycle and CI/CD tools is a top priority. Using automation, organizations can keep pace with the elastic, dynamic nature of cloud native applications and infrastructure.

Better Security, Better Apps

The tighter the integration between security and the CI/CD pipeline, the earlier threats can be identified, and the more the speed of delivery can be accelerated. Using the right cloud workload protection platform (CWPP) that seamlessly integrates with Jenkins, Bamboo, GitLab and others, DevOps teams can respond and remediate incidents even faster within the toolsets they use.

Hardening the CI/CD pipeline allows DevOps teams to move fast without sacrificing security. The automation and integration of security into the CI/CD pipeline transform the DevOps culture into its close relative, DevSecOps, which extends the methodology of DevOps by focusing on building security into the process.

As businesses continue to adopt cloud services and infrastructure, forgetting to keep security top of mind is not an option. The CI/CD pipeline represents an attractive target for threat actors. Its criticality means that a compromise could have a significant impact on business and IT operations.

Baking security into the CI/CD pipeline enables businesses to pursue their digital initiatives with confidence and security. By shifting security left, organizations can identify misconfigurations and other security risks before they affect users. Given the role that cloud computing plays in enabling DevOps, protecting cloud environments and workloads will only take on a larger role in defending the CI/CD pipeline, your applications and, ultimately, your customers.

The post A Better Way to Shift Security Left appeared first on The New Stack.

Kubernetes as a core technology has become foundational to modern application architectures and continues to expand its market presence. A recent survey of 500 full-time IT department employees conducted by Portworx by Pure Storage finds 87% expect Kubernetes to play a larger role in their organizations over the next two years, with 79% noting they have already increased usage of Kubernetes clusters over the last two years.

The primary reasons for increased reliance on Kubernetes cited by survey respondents are the need to increase levels of automation (56%), followed by reduced IT costs (53%), the need to deploy applications faster (49%), and digital transformation initiatives spurred by the COVID-19 pandemic (48%), the survey finds.

After initial adoption, many enterprise IT organizations quickly realize that Kubernetes is simultaneously the most powerful yet complex platform ever to be deployed and managed. Now those same enterprises are attempting to manage fleets of Kubernetes clusters that present even more networking and security challenges at levels of unprecedented scale.

While it may feel intuitive to run many workloads in a single Kubernetes cluster for easier management and better resource utilization, we observe an increase in the number of Kubernetes cluster deployments, whether it is a consequence of the development team’s own choices or for performance optimization of workloads running at the edge to be closer to users or to isolate workloads for organizational or legal reasons. 

Kubernetes was built by some of the world’s most talented software engineers for large-scale architectures. The issue is its complexity requires skilled software engineers which are a scarce resource across today’s highly competitive workforce. Kubernetes expertise is not only hard to find and retain, but the software engineers that have these skills also command some of the highest salaries in the IT industry. 

This exponential growth in deployed Kubernetes clusters coupled with the challenges of attracting and retaining in-house Kubernetes expertise leaves small-to-medium-sized IT organizations in a difficult position to keep up with Kubernetes clusters’ sprawl. The market needs simpler ways to industrialize Kubernetes growth at scale, whether through the help of a central control plane or automation, or both.

The Proliferation of Kubernetes Clusters Demands a Central Control Plane

As the fleet of Kubernetes clusters continues to expand, a central control plane is necessary to ensure that the system’s different components can work together effectively and efficiently. Without a central control plane, it would be difficult to manage and coordinate the different Kubernetes clusters, and to ensure that applications are running smoothly end-to-end. This also makes it easier for DevOps and administrators to have centralized management and control over the clusters. A central control plane also needs to take microservices networking into consideration by:

• Managing global and local traffic from one place while providing a dashboard overview of distributed environments,
• Applying settings such as traffic management rules and security policies globally across all clusters in a consistent manner and
• Providing a centralized Global Server Load-Balancing (GSLB) capability to increase reliability and reduce latency for applications spanning multiple regions in public and private clouds.

A centralized control plan with a simple-to-use web GUI is a convenient way to enable teams to quickly bootstrap projects. Organizations that are just getting started with Kubernetes will find this invaluable while they are still handcrafting individual cluster deployments.

But, as organizations accelerate their adoption and use of Kubernetes in production, manual management of multiple clusters becomes untenable.

Automation Is No Longer a “Nice to Have”

The only way to effectively navigate Kubernetes deployments at scale is to adopt the right automation and management tools. 

Organizations deploying Kubernetes must make it accessible to the small army of administrators that populate most IT teams. Most individuals seek full automation and audit-ability through GitOps — a version of DevOps automation — to deploy and manage infrastructure and applications across multiple Kubernetes clusters. 

At its core, GitOps promotes the use of declarative infrastructure and application definitions, which describe the desired state of the environment rather than the steps required to achieve it. Non-GitOps approaches and deployment strategies for provisioning clusters and deploying manifests are often fragmented and involve manual intervention, which costs engineers time and elongates the process of scaling.

GitOps solves the problem of managing and deploying infrastructure and applications in a consistent and repeatable way with easier collaboration (with full audit trails), version control and the ability to roll back changes. By leveraging GitOps-compliant tools, application teams take advantage of automating the self-healing, autoscaling and observability of Kubernetes clusters, as well as creating a consistent method for incorporating security and observability standards.

Regardless of the motivation behind the initial rise in adoption, it’s clear Kubernetes is now a permanent fixture in the IT landscape as clusters are increasingly deployed everywhere from the network edge to the cloud and everywhere in between. Investing in a GitOps-ready, central control plane will point organizations in the right direction of the next Kubernetes management frontier.

The post The Next Kubernetes Management Frontier: Automation appeared first on The New Stack.

Performance testing enterprise applications requires a high level of expertise as today’s applications are complex and require numerous supporting applications and systems. With every release, testers and developers need to ensure each outperforms the last and performance doesn’t degrade. But only a handful of experts within an organization can reliably measure the performance of a system as complex as the modern application stack.

And in today’s environment, the expertise needed to tackle an enterprise’s performance engineering is not always readily available. As a result, enterprises are finding quality assurance to be a bottleneck as they struggle to keep pace with release cycles. Or worse, experience outages that impact their business.

This predicament requires enterprises to consider how they are going to scale applications released daily with limited resources. Since personal expertise is difficult to scale, organizations can leverage technology to help, supporting the requirements for different types of teams, applications and deployment models. This approach not only empowers DevOps teams but fortifies and simplifies repeatability for project success.

What exactly does this approach look like? Where should enterprises start? And are there any larger considerations for implementation? Let’s answer these questions with five key aspects of a successful implementation for performance engineering at scale.

1: Promote Collaboration Between Experts and Non-Experts

Enterprise-wide performance engineering is a team sport. A highly collaborative approach fosters efficiency and effectiveness and enables performance expertise to scale without the need for additional experts. Developers, performance engineers, business analysts, and others working as a team make it easy to design tests with agreed-upon service level objectives (SLOs) that define measurable performance metrics. This ensures that everyone is measuring performance consistently and getting consistent results most efficiently. With consistent reporting, aspects such as root cause analysis and trend reporting are easier to generate across the board.

When operating effectively, performance engineering experts take on more of an enabler role. Instead of assuming responsibility for all testing operations themselves, they create the strategy and building blocks that allow non-experts to test at speed and scale. They can structure the test environment, implement quality-control guardrails, enhance delivery pipelines, and embed best practices into performance engineering processes that empower decentralized teams.

2: Automate Performance Testing in Continuous Integration (CI) Pipelines

Given today’s rapid development cycles, it’s impossible for performance engineers to manually build, run and analyze daily performance tests for hundreds of code pushes. These skilled engineers should instead focus on more strategic tasks. By integrating automated performance tests into CI pipelines as part of the pass/fail criteria, teams of developers can start testing APIs, microservices, and components at the beginning of the development process and continue testing throughout the evolution of the application.

3: Utilize No-Code/Low-Code Tooling

If different teams need to tailor their performance testing approach for their unique needs, testing must be user-friendly. Widespread adoption of testing tools correlates with ease of use and a shorter learning curve. Teams should seek tools utilizing no-code/low-code approaches that leverage intuitive drag-and-drop and point-and-click functionality to maximize adoption and ensure high quality.

In addition to testing through a codeless GUI, the platform should specifically enable DevOps teams to run tests in the command line interface (CLI) or within their day-to-day integrated development environment (IDE). It’s important to utilize no-code/low-code tooling to test fast, and release applications even faster. However, how quickly tests can be created is directly related to how easy the testing tool is to use, hence the need for tools with an easier interface.

4: Consider a Cloud Native Approach

Today, enterprises must ensure their approach to performance engineering is cloud-ready as more applications, processes and tools move to the cloud. An enterprise’s approach to performance engineering in the cloud should anticipate several complexities.

First, performance engineering tools should be vendor neutral so that performance and scalability can be measured across various cloud providers such as Azure, Amazon Web Services and Google. If one cloud provider has an issue, organizations need to have their applications ready to shift from one cloud provider to another quickly to prevent their users from noticing a change. While this is mainly for larger businesses, it is critical to understand your cloud service provider vendor lock-in impacts to ensure your business success.

Additionally, if a business is working on a migration initiative, whether that is to lift and shift to the cloud, re-platform, or refactor, organizations need to baseline performance before and then validate after the migration occurs. Despite its complexity, performance testing is crucial to perform alongside any significant changes.

5: Choose a Solution That Covers All Bases

In adopting a single performance engineering approach, an enterprise should first standardize on a performance testing solution that’s designed to support a wide range of enterprise testing requirements. A typical enterprise will employ a gamut of technologies and methodologies, carried out by either centralized teams of internal and external experts, autonomous development teams, or a combination of both. A standardized platform must work well for each of these groups equally. In addition, the chosen platform needs to test everything from enterprise-grade legacy monolithic core systems to dynamic microservices-based applications and packaged applications to ensure the migrations or transitions don’t have performance degradation.

When aligning on the solution, there should be a similarly wide technology coverage, from the newest frameworks to the tried-and-true technologies. Enterprises should be able to use the same solution to test individual APIs at the component level and the performance of all apps end-to-end. A standardized platform must work equally well for each scenario.

Finally, the chosen platform should not limit enterprises to a single deployment option. Virtually every organization’s environment is a combination of network types: on-premises and multiple public cloud providers. As enterprises increasingly modernize their business, they need a solution that can test performance for complex migrations. For example, moving SAP to an S/4 HANA implementation.

Today’s enterprises not only demand but expect the highest level of confidence in the quality of their software releases. While there are simply not enough expert DevOps professionals to keep up with the rapid pace of development, an agile performance engineering approach enables organizations to empower non-expert performance engineers to give enterprises the predictability, validation, scalability, and assurance they require. When companies create high levels of collaboration, utilize no-code/low-code tooling and automation, think cloud natively, and select solutions that can be utilized in all environments, they will ensure their business and critical applications are scalable for years to come.

The post 5 Tips to Achieve Performance Engineering at Scale appeared first on The New Stack.

Speed is a recurring theme in this episode of The Tech Founder Odyssey. Also, timing.

Eilon Elhadad and Eylam Milner, who met while serving in the Israeli military, discovered that source code leak was a hazardous side effect of businesses’ need to move fast and break things in order to stay competitive.

“Every new business challenge leads to a new technological solution,” said Elhadad in this episode of The New Stack’s podcast series. “The business challenge was to deliver product faster to the business; the solution was to build off the supply chain. And then it leads to a new security attack surface.”

Discovering this problem, and finding a solution to it, put Milner and Elhadad in the right place at the right time — just as the tech industry was beginning to rally itself to deal with this issue and give it a name: software supply chain security.

It led them to co-found Argon Security, which was acquired by Aqua Security in late 2021, a bit over a year after Argon started.

Though the pair had intended to “create a huge company,” as Elhadad put it, the timing proved right to sell to Aqua, the co-founders said.

As the industry moves toward creating a “golden path” for cloud native organizations, a wave of acquisitions is likely. “We see a lot of other companies making acquisitions of other kinds of startups,” Milner said. “And we see everybody in the chase of creating the unified experience for the customers.”

Elhadad and Milner, now Aqua’s general manager and senior director for software supply chain security, respectively, told the story of their startup journey and success to Heather Joslyn and Colleen Coll of The New Stack.

Listening to Customers

Argon began with the aspiring entrepreneurs doing their homework.

“We basically took a deep dive into this topic,” Elhadad said. “We start to look around attack surface trends in the market. We started speaking with different potential customers.”

Their market research revealed a massive potential customer base, he said, with CI/CD security — or lack of it — the biggest vulnerability: “It’s a huge thing … but no one was protected.”

As they set up Argon, Elhadad took the role of CEO, Milner became chief technology officer. They ignored friends’ cautions about starting a company during a global pandemic, reasoning that there’s never a perfect time to take the leap.

“Always there [are] disadvantages and advantages of the time that you’re doing it,” said Elhadad. “An entrepreneur [who] believes in a specific idea and has a strong team, you need to go for it, all in, no matter what. It will find the solution to fix the problem. Because [the problem] will show up every day.”

Elhadad offered some advice to other aspiring entrepreneurs, based on his and Milner’s experience with Argon: “Speak with customers from day one, learn how to do it, learn how to sell, and how to hear the customer, how to create a value proposition.”

Common Startup Mistakes

Both men had started companies previously. Elhadad’s, PaidIt, was purchased by YCD & Verifone; Milner’s, Datsmi, folded after a year. Today they are both investors in other startups, through YL Ventures.

They’ve learned a bit, Milner said, from working with early-days companies. Though a lot of those fledging organizations are comprised of very smart individuals, they often falter for very specific reasons, he said.

“The majority of them make the same mistake, especially technical founders, of falling in love with the solution,” Milner said. “So they often find kind of an unmet need, and they tackle the making of a solution. They start to code … and they get an amazing product. And the focus is on the solution itself.”

And then, he said, two potential issues arise. One is product market fit — either creating a solution for a problem that either doesn’t exist or creating a solution that can’t easily scale.

The other issue: “They have a hard time letting go. It was one of the mistakes that I feel I did in the previous company that I founded.”

Listen to the full episode to learn more about Elhadad and Milner’s journey, including how their leadership collaboration worked.

Correction: A previous version of this article mistakenly reported the amount Aqua Security paid for Argon Security. As an independent company, Argon received an offer from an unidentified investor of just under $30 million while seeking A series funding.

Like this episode? Here are more from The Tech Founder Odyssey series:

How Solvo’s Co-Founder Got the ‘Guts’ to Be an Entrepreneur

Feature Flags Are Not Just for Devs

After GitHub, Brian Douglas Builds a ‘Saucy’ Startup

The post How 2 Founders Sold Their Startup to Aqua Security in a Year appeared first on The New Stack.

To automate deployment processes, CI/CD needs context: deployment configurations, build configurations, artifacts, version numbers, dependencies, environment variables, test results and more. This data doesn’t exist in one place — it is usually scattered across multiple systems and tools.

For example, deployment configurations might be stored in a separate YAML file, environment variables might be defined in a script or in the deployment manifest and version numbers might be tracked manually in a spreadsheet.

Too many sources of truth can lead to several problems, including increased complexity, metadata inconsistency, difficulties updating data and, most of all, an inability to apply automation. Software catalogs, the core of internal developer portals, can provide a solution.

First Step: A Software Catalog That Can Store CI/CD Data

The first step is creating a software catalog with the right data inside. It should be a general-purpose software catalog that allows adding data types with different properties and relationships, providing flexibility to enable everyone to bring their own data model to the catalog.

An internal developer portal is at the core of platform engineering. It presents developers with the self-service actions built as part of the platform and also with a software catalog.

This is where it gets interesting. From the developer experience point of view, the software catalog can be explained as a redacted, whitelisted data store that is curated to help developers overcome cognitive load (as an example, see how K8s data can be presented to developers).

But that isn’t the entire story. Actually, using the software catalog for CI/CD is very powerful. A software catalog can store data about builds, environments, cloud and a lot more. This type of software catalog can be highly beneficial to create a single source of truth for CI/CD context.

Port is a platform for building no-code, holistic, Internal Developer Portals. Port’s software catalog covers microservices, resources, custom assets and fits any data model, with in-context maturity scorecards. Its portals support any developer self-service action and workflow automation.

Learn More

The latest from Port

Platform engineering teams we’re talking to are actively realizing these benefits, especially with regards to CI/CD metadata. They use the software catalog as a single source of truth for CI/CD, and are also using the CI/CD data in the software catalog as part of their automated workflows.

By including relevant data about the clusters, environments, cloud regions and vendors in the software catalog, the CI/CD process can be more intelligent and automated, leading to better engineering. It decouples CI/CD from the contextual data it needs, separates controls and makes it easier to troubleshoot failures and broken pipelines.

Through the developer portal, these capabilities also help platform engineering teams provide developers with better visibility into the deployment process, as they can see the deployment status and any errors that occur in real time.

Next Step: Version Control and Security

Once the software catalog is set up, the benefits of one source of truth for CI/CD data can be taken even further when it’s also used for version control and security,

Tracking all the changes made to the metadata and configuration files improves the traceability of metadata changes over time. This can be useful for auditing purposes and for understanding the evolution of the deployment process.

Additionally, it drives better collaboration (with version and change tracking), faster issue resolution and the ability to quickly revert to a previous version and improved compliance. When CI/CD data is fragmented — think of scattered version history in git — it’s difficult to do this, but it’s much easier with the software catalog.
A software catalog usually ensures that only authorized users can access and modify the metadata, reducing the risk of unauthorized access, data breaches and other security incidents. Examples are a misconfiguration that results in making an S3 bucket public or exposing a service with personally identifiable information to the internet.

How It Works

The software catalog is essentially a centralized database that stores all the metadata related to the CI/CD process. It can be accessed and modified through a REST API, which enables CI/CD pipelines to interact with the metadata store programmatically. Data types, properties and relationships can be easily added when needed, since different organizations do DevOps differently.

What data should be accessed and stored? This depends on what we call your data model, meaning the properties and categories that are important within your pipelines. For instance:

• You can organize the catalog by different categories, each containing metadata related to a specific aspect of the CI/CD process. For example, there might be a category for deployment configurations, a category for environment variables and a category for version control.
• Within each category, there would be different metadata items or keys. For example, within the deployment configurations category, there might be metadata items for the deployment target, the deployment strategy and the deployment version.

The CI/CD pipelines can interact with the metadata store by using a REST API, specifying the category and metadata item they want to access. For example, to retrieve the deployment target for a specific application, the CI/CD pipelines might send a GET request to the deployment configurations category, specifying the metadata item for the deployment target.

The Importance of Graph Databases for Software Catalogs

Graph databases come in handy for software catalogs. Since the different entities in the software catalog have complex relationships (for instance, a service is deployed on a namespace in a K8s cluster in a cloud account) and those relationships are important, you need the ability to natively query them. A graph database lets you do just that. This is particularly useful in the context of a CI/CD pipeline, where developers, DevOps and machines need to be able to quickly access information about how different parts of the system are related.

• Or let’s say we want to identify all the services that use a particular image version. Without a metadata store, you will need to manually search through various services’ configurations and documentation to find the ones that match. But with a graph database, we can create nodes for each service and link them to the image version they use. This allows us to quickly query the graph to find all the services that use the desired image version. We can start by querying the image version node and then traversing its relationships to the service nodes. We can even add additional information to the nodes, such as the environment the service is running in, the date it was last updated and any associated alerts or issues. This provides a comprehensive view of the entire system and allows us to easily track and manage our services.

• For example, let’s say we want to identify all services running in a particular region (for instance, if you’re operating a large-scale cloud platform, serving customers across different regions). Without a graph database, we would need to perform multiple queries across different data sources and try to piece together the information. However, with a graph database, we can do it in one query.

This ability to natively query complex relationships is critical in enabling developers and machines to perform impact analysis, manage configurations, run continuous tests and manage releases more effectively. This not only simplifies the CI/CD process, it also helps to ensure the overall stability and reliability of the system.

Software Catalogs Need to Be API-First

Now we need to think about how to easily get data into the software catalog. Easily ingesting data into the software catalog requires an API-first approach. This includes data from cloud providers,Kubernetes (for cluster data), git providers, Infrastructure- as-Code (IaC) tools such as Terraform or Crossplane and more.

An API-first approach also makes it easy to build integrations with other tools and systems, such as creating a dashboard with information about your infrastructure and applications. This can help you build a more comprehensive and useful metadata store that provides a holistic view of your infrastructure and applications.

Conclusion

The rise of platform engineering and the internal developer portals that are used as a core interface for developers also presents an opportunity to create a software catalog that can be useful not just for developers. A software catalog with CI/CD metadata can create a single source of truth, solve version and security issues, and allow automation of deployment processes and more. To see what a general-purpose software catalog can contain, go to Port’s live demo here.

The post Simplify CI/CD with a General-Purpose Software Catalog appeared first on The New Stack.

The Software Bill of Materials is now essential for program security, and IBM is helping by donating two open source projects to the Open Web Application Security Project.

IBM has contributed two open source projects, SBOM Utility and License Scanner to the Open Web Application Security Project (OWASP). There they’ll be integrated into CycloneDX, OWASP’s flagship Software Bill of Materials (SBOM) standard.

The SBOM Utility is an API platform designed to validate CycloneDX or Software Package Data Exchange (SPDX) format SBOMs against their published schemas. It can also help validate derivative, “custom” SBOM schemas that require stricter data requirements. It can evaluate and report on component software, service, and data license information to assist organizations in risk evaluation against configurable usage policies.

SBOM Utility and License Scanner

SBOM Utility can also be used to run queries against CycloneDX SBOM contents with regular expressions (regex) and generate custom reports. Additionally, it supports the work of the OWASP Software Component Verification Standard (SCVS), which is defining a BOM Maturity Model (BMM) to help identify and reduce risk in the software supply chain. Finally, it can be used to process Vulnerability Disclosure Report (VDR), and Vulnerability Exploitability eXchange (VEX) documents.

The License Scanner, on the other hand, scans files for licenses and legal terms. It can be used to identify text-matching licenses and license exceptions from the complete published SPDX License List. License Scanner is designed to be integrated into existing SBOM software, Continuous Integration/Continuous Delivery (CI/CD),  or used by itself as a command-line utility. The tool scans the license text against the set of SPDX license templates and returns the CycloneDX LicenseChoice data incorporating three ways of expressing licenses: SPDX License ID, SPDX License Expression, and License name.

IBM had been using the License Scanner in its own CI/CD service, IBM Cloud’s Continuous Delivery, and DevOps toolchains. It’s also been used in IBM’s legal clearance process for open source and corporate software. In short, it’s battle-tested and ready for deployment.

Both tools are written in Go to take advantage of the language’s built-in typing enforcement and memory safe features and its ability to be compiled for a wide range of target platforms and architectures and be compatible with cloud native platforms.

Andrew van Der Stock, OWASP’s Executive Director, thanked IBM for these donations. Van Der Stock said, “validation against defined schemas is an important integrity control, and the ability to scan code to identify licenses is critical for some use cases. For many mergers and acquisitions, being able to rapidly and accurately identify licenses present in code make or break deals.

So, these donations will help not only developers looking to secure their code with SBOMs, but they’ll also help the top brass with their mergers and acquisitions. This is a pair of open source donations that will help everyone.

The post IBM Donates SBOM Code to OWASP appeared first on The New Stack.

These days, if you are producing software as containers, you have someone downstream scanning your container images for vulnerabilities, and that makes your job harder than it was in the past.

The world has plenty of vulnerability scanners, many of them great. What it doesn’t have is a scalable way to manage and remove vulnerabilities. Specifically, developers need a way to automate the process of identifying and removing vulnerabilities in container images during the build process, and DevSecOps professionals need a way to track and communicate what they’re fixing to every player in the ecosystem up and down the software supply chain.

Modern security depends on this, and right now, the process is broken.

Under the current climate of software supply chain concerns, organizations and development teams are struggling to keep up with managing CVEs (Critical Vulnerability Events) and are drowning in CVE noise. Questions they face include, “Which vulnerabilities are exploitable, and which ones aren’t?” and “How do we prioritize what to remediate first so that we’re effectively deploying our limited engineering resources?”

How do teams even make a dent in the problem when the average container has 284 vulnerabilities and companies are running hundreds or thousands of container images in their infrastructure at a time? According to our 2022 Container Report, four new vulnerabilities appear for every one that gets remediated.

The root of the problem is that what makes a container ideal for development makes it vulnerable in production. Tools and packages in the development stage make developers’ lives easier. When you move to production, however, you want to remove anything that doesn’t need to be there, because it inflates the attack surface and introduces potentially vulnerable packages that you simply cannot have in production.

My co-founder Kyle Quest and I use the mantra “ship only what you need to production” to describe what we’re trying to accomplish at Slim.AI. For the companies using our platform, that means creating a system for removing CVE noise automatically. It removes the vulnerabilities associated with software that’s not necessary for an application to run. By reducing this noise, we help teams prioritize vulnerabilities that do matter.

Helping Developers Reduce CVE Noise

This process of making containers ready for production is a profoundly difficult task for most developers. Vulnerability scanners are valuable and necessary tools to help secure your software, but having a strong risk reduction posture comes at a cost — namely, developer productivity.

In fact, according to our recent survey, when developers were asked “Which tasks are adding more work to your job?” — finding and removing vulnerabilities from containers was the number one task adding to their workload.

Organizations today are working to improve the signal-to-noise ratio on vulnerability scanning and to prioritize security fixes as efficiently as possible. At Slim.AI, we are working on ways to automate the reduction, management and sharing of vulnerabilities provided by scanners while also removing code that is not needed in production.

Current Solutions Are Necessary, Not Sufficient

Fig. 1: Tools to discover, optimize and manage containers pre-production leave a huge gap for developers to bridge on their own. Slim.AI is solving that problem.

There are several different approaches to addressing the need for vulnerability remediation, all with benefits but also with significant drawbacks.

Manual container optimization approaches like Alpine images start with a developer choosing a minimal base image and then adding all the tools they need for development. For production, they — hopefully — back those tools out of their Dockerfile and run it through a vulnerability scanner so the containers can be released to production.

If you’re an experienced backend developer or DevOps pro, you know both the need for this process and how to do it — without breaking anything. But as your organization begins to expand its adoption of cloud, these teams of specialized “developer experience engineers” or “cloud architects” can become hard to scale, as they’re expensive, rare and in high demand.

Additionally, many organizations have neither control over nor visibility into the base images being used by the vendors or upstream open source projects. And with new base images often come new operating systems that can add overhead to debugging low-level system issues with DNS and kernel libraries.

Vulnerability scanning, on the other hand, is a reactive or point-in-time solution. There is a long and growing list of vulnerability scanners, and many do a great job. The problem is, they don’t solve the root problem, because when you get a report of 200 to 300 vulnerabilities, perhaps even 1,000, in a given container, with hundreds or thousands of unique containers, that’s simply not actionable. It’s too much to reasonably address. Plus, you know that most of those vulnerabilities are irrelevant to the application you’ve built. Again, this approach is unscalable, because the signal-to-noise ratio is poor.

Another aspect of this problem is that vulnerability scanners can provide a false sense of security. There are plenty of aspects of “attack surface” that have nothing to do with CVEs. A “zero-vulnerability” image can still be exploited in certain conditions using things like shells, package managers, or tools like git and curl.

Policy engines are a powerful tool because they don’t allow anything into production that doesn’t meet criteria set by the DevOps or DevSecOps team. Examples might be if code or artifacts aren’t signed or if critical vulnerabilities rise above a certain threshold. Policy engines, like the Cloud Native Computing Foundation–sponsored Kyverno, are a good thing — even a critical thing — but they act like a wall: they block developers from shipping code to production and siphon their time away from coding into managing lengthy break/fix cycles.

It is my belief that successful organizations will find ways to push compliance upstream, putting the necessary tools in the hands of developers to adhere to their policies, in addition to having the requisite downstream checks to prevent security issues in production.

The Slim.AI Difference: A Developer-First Approach

Slim.AI recognizes the shortcomings of these current container security approaches and has built a platform for developers that automates the process of shipping slim, secure containers. The benefit of the Slim.AI approach is that it gives developers the freedom to use any image they want for development. With our tools, developers can use whichever base images and tools they prefer, then run their container through our automated process to generate a new, production-ready container, guaranteed to adhere to container best practices.

Additionally, the Slim.AI platform for proactive vulnerability reduction works with third-party images, so you can harden any images you are consuming from open source projects or third parties (or request they run their containers through Slim prior to shipping them to you). For example, we’re working with leading security provider BigID to harden their container images and provide their teams with valuable reports on the security profiles of their containers that they can share with customers.

With the Slim.AI platform, there’s no new code to write nor a new ecosystem to learn. You simply run our software in our cloud or in your existing infrastructure and automatically add a new layer of container security and visibility to your organization. Your security teams get the peace of mind that containers are adhering to security best practices (and automated reports to prove it), and developers can spend more time building awesome features and less time chasing down patches to random open source libraries.

Fig. 2: Slim.AI is actively removing vulnerabilities as part of an automated process that integrates with the tools and processes developers are already using.

Conclusion

At Slim.AI, we believe that successful organizations will be those that take the long view on software supply chain security with their containers, by automating vulnerability triage and remediation,  reducing software attack surface and securing images.

We’re adopting the position that automation can empower developers. Supply chain security is such a critical issue that we need a solution that scales, and automating the security process in development is the only way to solve it.

By implementing the Slim.AI platform into your development workflow, your security teams benefit by knowing developers are taking every precaution to secure their containers in an automated way. And developers benefit from the tools, insights and automated processes to harden their containers, eliminating guesswork and “best effort” practices.

We’re bringing on new design partners every day to build the best platform we can, and we hope you will join us in doing so.

The post Slim.AI: Automating Vulnerability Remediation for a Shift-Left World appeared first on The New Stack.

By 2017, Uber’s codebase was incredibly fragmented to the point where problems would bleed into library versions, build tools, dependency management and collaboration, and code sharing was deeply affected. So the company moved developers to a remote development environment, based on a monorepos run on a Kubernetes cluster.

Uber’s build-strategy team recommended migrating to a monorepo architecture to host Uber’s code. Their plans encompassed over 10 programming languages, over 4,000 services, over 500 web apps, more than nine build tools and more than six configuration tools distributed across thousands of repositories at the time, with trunk-based development and one single version per third-party library at that time.

Uber’s recent blog post details the company’s custom in-house remote development environments, Devpods, which run on Kubernetes clusters. These secure environments need zero set up, and they run in the cloud rather than on laptops.  Build times have improved, and over 60% of engineers are using Devpods as of November 2022.

The new monorepo, a single centralized build platform (built on Google’s Bazel), brought better dependency management, consistent universal production library versions, easier support for a standard set of tools and processes, and improved visibility, collaboration and code sharing.

But it also brought challenges to the daily code edit-build-run development loop — development was taking longer, much longer, with cloning taking several hours in some cases.

Remote development challenges plus monorepo challenges led to the need for innovation.

Port is a platform for building no-code, holistic, Internal Developer Portals. Port’s software catalog covers microservices, resources, custom assets and fits any data model, with in-context maturity scorecards. Its portals support any developer self-service action and workflow automation.

Learn More

The latest from Port

Platform Engineering

The idea of Devpod is to allow developers to build in the cloud, using a faster machine, with all the tooling kept in a secure, controlled environment.

A Devpod is tailored specifically for an Uber developer’s needs and what Uber considers the “best possible developer experience.”

Devpods run as a container on a Kubernetes cluster, with the necessary tooling and compute resources to enable seamless development on Uber’s large-scale monorepo. Productivity is enhanced by Devpod’s improved performance and nearly zero setup. Security was improved by running the latest stable versions of each dependency.

Devpods are persistent, meaning engineers won’t lose their setup, files or code changes while working on different devices or collaborating with other engineers in a single environment. They use cloud resources — up to 48 cores and 96GB of RAM over laptop resources, which allows for faster Git, Build, and responsiveness. Devpods come with the latest version of IDEs configured with cached IDE index. Git configuration is optimized, and the Linux file system is used rather than the laptop file system, which provides a better performance environment.

Since Devpods run in the cloud computing environment, there’s less freezing/overheating, better battery life, and developers are able to freely switch between environments.

Other advantages include:

• Zero Setup and Maintenance: Devpod has all the tools and configurations needed to start working in any of the main monorepos right away. Tools are ready to go. Environments are provisioned on demand, IDEs are set configured, repositories are pre-cloned.
• Secure: Devpods are updated automatically with the latest tooling and security upgrades overnight. Since Kubernetes pods run in the cloud, and don’t have laptop battery constraints, it’s “trivial” to scan disk for malicious artifacts or activity during off hours. Vulnerable applications can be patched during non-work hours as well.
• Prioritizing Low Latency: Devpods are deployed globally with the goal of reducing latencies for developers and avoiding the “sluggish” feeling for UI operations.

User Experience

Developers are presented with a “flavor,” a Docker image preset with the tools and default settings and configurations for a specific group of engineers. The following flavors are supported:

Putting the tooling and configuration into the docker image “made perfect sense,” according to Uber engineers. They built the Devpod image base on top of the production image base to reuse their critical production software configuration. Uber uses Debian for production workloads. The consistent base allows for the current infrastructure to resume for the building and distributing of internal packages.

This web-based interface was created to provide a better first-time experience with Devpods and improve troubleshooting. The future goal is to extend Devpod Central and bring visibility to resource usage and consumption.

The following IDEs are available in Devpods: VS Code remote, VS Code over web, JetBrains IDE via ssh using JetBrains Gateway Vim and Emacs via terminal.

Kubernetes as a Platform

Moving from laptops to the cloud was a great start, and Kubernetes provided additional necessary building blocks. The ability to host containers on powerful hardware, networking to connect containers, and persistent volumes to store engineering work between restarts was also a necessary step.

Uber used custom resource definitions (CRD), a way of extending Kubernetes by providing a data structure stored inside the control plate, and removed the need for an internal database for Devpod metadata. It also allowed for the creation of and access to Devpods using kubectl.

Kubernetes’ CRDs also have the ability to write custom software that reacts to resource changes, called the operator pattern. Uber combined custom resources and small custom controller to translate a high-level Devpod description into standard Kubernetes primitives that can be handled by standard Kubernetes controllers.

The custom controller was later extended to react to VolumeSnapshotContents and PersistentColumeClaim in order to cut costs when Devpod isn’t used.

Progress So Far

Devpods are effective in reducing the local Git command performance. The chart below shows that git status times have been consistently below four seconds.

Devpods provides a 2.5x improvement over laptops on average for longer, more complex builds and 1.8x improvement over laptops for average builds.

As of November 2022, over 60% of Uber software engineers have adopted Devpods.

Devpods have a bright future at Uber. The goal now is to make Devpods completely seamless for Uber engineers. Some areas of exploration are reduction in the time it takes to set up Devpods, ephemeral Devpods and seamless file transfer between laptops and Devpods.

The post DevPod: Uber’s MonoRepo-Based Remote Development Platform appeared first on The New Stack.

More application teams are adopting continuous integration/continuous Delivery (CI/CD) workflows to facilitate application development, which means their organization needs to deploy automated and integrated security to defend these workflows.

Organizations trying to accelerate innovation of their cloud applications and services place CI/CD pipelines at the heart of everyday operations. When configured correctly, they provide visibility into how software is being developed and automate manual operations to guarantee consistency across delivery processes. A network’s infrastructure has access to a variety of resources, such as analytics keys and code signing credentials, through these pipelines.

DevOps teams have used containers and container orchestration platforms like Kubernetes to build and deploy their applications because the majority of contemporary apps are built using a microservice architecture. Consequently, any solution that aims to safeguard CI/CD workflows must include container security as a key component. The five factors listed below can help DevOps teams make sure their container strategy isn’t jeopardizing security.

1. Automatically Scan Source Code

To get a precise assessment of your application’s vulnerability, do thorough and in-depth scans. Embedding continuous application security testing into the process of creating and delivering apps in the main DevOps CI/CD environments will help you assess and identify security flaws that you can patch or eliminate during the software development life cycle (SDLC).

The Fortinet Security Fabric platform delivers broad, integrated, and automated protections across the entire digital attack surface. Ranked #1 for most security appliances worldwide, more than 580,000 customers trust Fortinet to secure their business and accelerate their digital journey.

Learn More

The latest from Fortinet

2. Set up Continuous Runtime Security 

Securing running microservices is just as crucial to an effective CI/CD security solution as is preventing application breaches by moving security to the pipeline’s earlier stages. The context necessary to comprehend Kubernetes structures — such as namespace, pods and labels — is not provided by conventional next-generation firewalls (NGFW). Once the perimeter has been compromised, the risk of implicit trust and flat networks on thwarting external attacks provides attackers a great deal of surface. As a result, it’s important to leverage a platform that enables continuous security and centralized policy and visibility for efficient and effective continuous runtime security.

3. Seamlessly Plug Security into the CI/CD Workflow

The majority of application teams automate their build process using build tools like Jenkins. Security solutions must be included in popular build frameworks to bring security to a build pipeline. Such integration enables teams to pick up new skills quickly and pass or fail builds depending on the requirements of their organization. For instance, a policy needs to be set up to fail builds when a critical vulnerability is discovered in an image if an enterprise has a security requirement that forbids the deployment of an application with critical vulnerabilities.

4. Build Images with Security in Mind

Third-party libraries and source codes are frequently used in images. It is essential to parse libraries and packages before creating an image to produce a complete report of all vulnerabilities (CVEs) and the libraries/packages where vulnerabilities are found. If particular libraries might create a security risk, they should also be excluded. A vulnerability report might also be able to reveal whether an image contains credentials or other sensitive information.

5. Use CIS Benchmarks to Run Compliance Checks

Running static tests to find potential vulnerabilities in systems using container orchestration platforms like Kubernetes is now essential as those platforms become more widely used. It’s a good idea to follow the recommendations for Kubernetes’ best security practices made by the Center for Internet Security (CIS). This provides suggestions for setting up Kubernetes to maintain a robust security stance, such as blocking anonymous API server queries and only allowing non-root users to run containers.

Better and more effective DevSecOps

Broad, comprehensive cloud security solutions that are natively integrated across major cloud platforms, along with a security fabric approach, are key to securing digital acceleration of teams’ application journeys. Solutions like the Fortinet Cloud Security portfolio (example shown above) empower organizations to achieve reduced operational complexity, greater visibility and robust security effectiveness with consistent policies across all hybrid and multiclouds, centralized management, deep visibility across applications and workloads.

The post Top 5 Considerations for Better Security in Your CI/CD Pipeline appeared first on The New Stack.

Kubescape’s official acceptance this week by the Cloud Native Computing Foundation (CNCF) as a sandbox project represents the beginning stage in the journey to offer a comprehensive open source security platform for Kubernetes projects, the project’s creators from ARMO say.

According to Kubescape’s documentation, the open source Kubernetes security platform covers the gamut of the lifecycle of applications and their updates for Kubernetes applications. This includes IDE, CI/CD pipelines and clusters for risk analysis, security, compliance and misconfiguration scanning.

The key operative words are “platform” and “Kubernetes.” The platform part means that Kubescape is not just another security tool with very specific functionalities for Kubernetes among legions of alternatives. The Kubernetes part is essential because this means that the platform is for Kubernetes only.

Kubescape is used to integrate with the long checklist of the necessary tools your DevOps teams would like to add for use with the platform, such as for software-bill-of-materials (SBOM), signature scanning and policy controls. It begins running its scans at the very beginning left end of the production cycle and extends across CI/CD and throughout the deployment and cluster-management process.

Used to find and fix misconfigurations and vulnerabilities across such: frameworks as NSA-CISA, MITRE ATT&CK and the CIS Benchmark, Kubescape scans YAML files, and Helm charts and clusters upon deployment. Kubescape can also be integrated with Jenkins, CircleCI, GitHub Actions, GitLab, IDEs (i.e. Visual Studio Code) Prometheus, Lens and Docker.

“We want to be the CNCF’s open source Kubernetes security platform; that’s my vision. We want to consolidate Kubernetes security into a single platform,” CEO and co-founder of Shauli Rozen of ARMO, told The New Stack. “I really think this is something that has been missing in this space.”

The concept of an open source, CNCF-donated security platform exclusively targeted for Kubernetes is appealing. But more remains to be seen as to how this open source project is adopted, Torsten Volk, an analyst for Enterprise Management Associates (EMA), told The New Stack.

ARMO also now offers ARMO Platform, as an additional security layer on top of Kubescape. It provides what the company calls a “ready-made” security platform for Kubernetes for SaaS or on-premises deployments. It can be deployed on hosted Kubernetes platforms including Amazon’s Elastic Kubernetes Service (EKS), Microsoft’s Azure Kubernetes Service (AKS), Google Kubernetes Engine (GKE), Red Hat OpenShift and others.

Cogs and Wheels

Kubescape largely relies on Open Policy Agent to verify Kubernetes objects against a library of posture controls. In Kubescape’s documentation, monitoring results are printed and can also be:

• Exported to JSON or junit XML.
• Rendered to HTML or PDF.
• Submitted to a cloud service.

Meanwhile, the company plans to open source a number of proprietary features and open source Kubescape’s backend code during the coming quarters for KubeScape, Rozen told The New Stack. The features it plans to open source include widening the process of continuously monitoring the runtime elements and “making sure that they’re not being changed,” in the event of a memory attack, for example, he said.

Meanwhile, in order to win over developer, security and operation team members, Kubescape must be able to demonstrate it can seamlessly fit into their current way of working and enable all Kubernetes-related personas to benefit from security guardrails and best practices sourced from the Kubernetes community, Volk said.  “This could finally give companies a leg up in the eternal race against the bad guys,” Volk said.

There are two categories of customers that Kubescape user customers typically fall under. These include large organizations that made the shift to cloud native but continue to maintain investments in other types of infrastructures outside of the Kubernetes sphere. The other end of the spectrum consists of recently created organizations that maintain “very dedicated Kubernetes environments,” Rozen said.

The all-Kubernetes organizations, mainly consisting of small- to medium-sized companies, is our sweet spot to be honest at the moment,” Rozen said.

The post Kubescape: A CNCF Sandbox Platform for All Kubernetes Security appeared first on The New Stack.

“There is a problem with the cybersecurity industry,” said Ory Segal, Chief Technology Officer of Prisma Cloud, Palo Alto Networks (PAN), at the company’s Ignite conference in Las Vegas late last year. “And that’s the short attention span that we have. New technologies sway us from our main goal. Our main goal is to secure the applications. If you think about it, our customers are building applications. They’re not in the business of building clouds. So selling them cloud security is not what we should be doing. We should be selling them application security.”

Customers secure applications, but there’s a looming issue that developers face., namely insecure software supply chains with vulnerable third-party dependencies. The application security environment looks more like a network with an API-centric architecture and multiple layers needing protection.

“If we looked at applications back at the end of the 90s, the beginning of the 2000s, we had the physical infrastructure, we had the network, the app and the data layer,” Segal said. “And applications look really simple. We have the web servers, web backend, database, That’s pretty much it. It was very easy to secure these applications. You would place a wire from the perimeter or firewall and that’s pretty much it.”

Today, there are thousands of application security point tools. The PAN goal is to help customers consolidate into an end-to-end approach, which PAN calls code-to-cloud. The objective: offer a platform to secure cloud native architectures built on microservices dependent on multiple APIs.

Segal said that attackers know how to traverse the applications to access the data. They will start through the application layer, a web form, for example, that stores the file in the cloud. An API call will get sent from the application layer to the workload, which will execute its logic and pull the file from the storage bucket, allowing the leaking of information.

Inspect the Packets

Point tools provide microscopic views, looking at every point in the chain. There might be a tool for the API calls or a tool in front of the web application firewall (WAF).

There are two alternative approaches to point tool approaches, Segal said. A service provider may take the plumbing approach and pull data from different vendors to create overlay insight.

“I think it has more cons than pros from a security vendor perspective,” Segal said. Mostly because you have to rely on signals from others, their quality, their format, you have to continue updating the way you consume the data. It is easier to develop.”

PAN follows the “one ring” approach, Segal said. It allows PAN to control the platform and the underlying analysis. In turn, that provides better security as the vendor owns the signals, in this case, PAN.

“Yes, it’s harder, because you have to have the technology and the knowledge and the know-how to build all these analysis centers,” Segal said. “But the benefits you’re getting from that as a security vendor are tremendous.”

“Prisma cloud is the platform that helps to secure cloud native applications,” Segal said. “It’s not a cloud security platform. I know the name is misleading. It is a cloud native application protection platform, and it provides different modules from the left to the right — from detecting vulnerabilities, secrets, infrastructures, code problems, all the way to the right where we provide the web API security module that prevents attacks against APIs.”

Integrated Security

Scott Moser, chief information security officer at Sabre Technologies, a long-time software provider for the travel industry, said onstage at the conference that already integrated security tools are better for Sabre than working across multiple tools and their respective APIs.

“I’d rather have my team using the security tools than spending their time integrating those security tools,” Mosier said.

Segal said the tools teams use often vary for application security, depending on the layer in the application stack. There is further complexity with the dynamic nature of CI/CD environments and the constant addition of new tools. The process is so fast that code may take just minutes to go from repository to production.

PAN’s acquisition of Cider Security illustrates the company’s shift-left approach. Cider is a platform for software supply chain security. The Cider Security service gives a developer views into the infrastructure and tools that developers use in CI/CD to determine the risks they pose. Code is not the focal aspect for Cider; it knows the myriad of tools developers use.

A choice of tools for CI/CD offers attackers ample opportunities to attack. With all the tools come risks to the code. Developers need more visibility into the tool configurations. Cider Security mitigates these kinds of risks. It serves as an overarching element that integrates with the customer’s solutions.

By default, the entire CI/CD workflow is programmable, which provides Cider a way to give PAN integrations for platforms such as Jenkins, GitLab or GitHub. By offering deeper integrations, PAN provides the ability to manage complexities when dealing with engineers.

The Cider technology still needs to integrate into Prisma Cloud.

The platform approach to application security is gaining moment, said Fernando Montenegro, a senior principal analyst with Omdia out of the Toronto area. He said PAN competes with the likes of Trend Micro, CheckPoint, and Rapid7. Competition is increasing from companies such as Sysdig, Lacework, Snyk, Orca, and others, for example, that offer continuous delivery tools that integrate with security tooling.

Montenegro said there are some nuances to consider, such as can a customer get by with an offering from an existing vendor that they use for CI/CD or some other aspect of application security.

In a Twitter thread, I asked people about CI/CD security, its complexity, and how to secure it. Here’s a look at what people had to say.

I am researching CI/CD security and it just seems so complex, considering the different languages and frameworks people use. Does anyone have any approaches they are following or work by that looks at the issue from a platform perspective to examine all the layers?

— Alexander (@alexwilliams) January 3, 2023

The post The Main Goal: Secure the Application Workload appeared first on The New Stack.

Gitlab is reinventing its Web IDE to better support remote development in cloud-based environments.

The new Web IDE became available in beta for everyone last month and is already enabled by default on GitLab.com. It has received positive, important feedback less than a month after its release, said Eric Schurter, principal product manager for Create at GitLab.

“One of the reasons we chose an open beta approach is that we wanted the community to help us understand which features are most important to address immediately,” Schurter told The New Stack. “We’ve already received incredibly valuable feedback and have implemented some improvements.”

New Features Already Planned

Based on user feedback, Gitlab plans to iterate on the committing and code review experiences, and add some more significant features, such as being able to install and run third-party extensions in the Web IDE and to search across an entire project.

The previous Web IDE was based on the open source Monaco editor, but didn’t compile code and wasn’t customizable to a developer’s own workflow, GitLab said in announcing the beta. The new web IDE is built on the Visual Studio Open Source project and designed to meet developers where they are, Schurter said.

“Visual Studio Code is one of the most popular IDEs out there and bringing a familiar experience to the browser, right inside the GitLab UI, means developers can be more productive and more confident making changes, without taking the time to switch contexts in their local development environment,” he said.

The Web IDE also builds on what VS Code offers, adding (for example) custom extensions that handle tracking changes to the file system, committing those changes to the repository using the GitLab API, and establishing a connection to a remote development environment, he told The New Stack.

“We’re excited to leverage the benefits of frequent upstream contributions, contribute back to the project whenever possible, and implement GitLab-specific functionality as needed,” he said.

More Cloud Development Support in IDE

Cloud development — or as GitLab terms it, remote development — is a key focus for the beta Web IDE. Developers spend hours managing and updating their local dependencies or re-installing package managers to troubleshoot conflicts, Schurter said. Larger teams can spend days, or even sometimes weeks, onboarding new developers to a project, he added.

“By defining a stable, reproducible development environment in code, developers will be able to create ephemeral instances of these environments in the cloud and be ready to contribute from the Web IDE or their local IDE within minutes,” he said. “There are other offerings with similar functionality, but the real benefit will be having it all contained and managed within GitLab’s single DevSecOps platform.”

GitLab team members have been using the beta Web IDE to update the GitLab handbook, said Michael Friedrich, senior developer evangelist at GitLab.

“I’m excited about the direction with remote development to ensure that everyone can contribute,” Friedrich said. “This will solve the limitations with local hardware and provisioning the development environment, and make teams and open source contributors more efficient.”

The Developer Experience

He provided an overview of the developer experience, saying the Web IDE is integrated into the GitLab DevSecOps platform and can be accessed directly from common workflows: Start a new project, edit an existing file, or open the IDE from a merge request to address review feedback.

“The Web IDE file explorer provides access to all files in the repository to make changes, with intelligent auto-complete suggestions and syntax highlighting,” he said. “Once the changes are committed to a new branch, a new merge request can be created. The merge request will trigger CI/CD pipelines, security scans, review app deployments, and observability for fast feedback in the DevSecOps lifecycle.”

Markdown preview comes with it, which can help with documentation, he added.

Frontend and web developers should know that they quickly can make changes to the web app source code and deploy them using a merge request, with feedback in review apps, or they can use continuous delivery with production changes, Friedrich said. The CI/CD pipelines run automatically after persisting the changes in the Web IDE. Developers can start a live preview with a development web server from the terminal when remote development environments are available with on-demand cloud development environments, he added.

Features for Veteran and New Developers

Advanced users can take advantage of the extensibility of the Web IDE, Friedrich said, adding that in the future, users will be able to install VS Code extensions into the Web IDE for a more customizable experience.

Another planned feature that Friedrich said will benefit developers and DevOps engineers: The GitLab workflow extension for VS Code natively in the browser with direct CI/CD pipeline feedback. That said, the IDE’s lightweight approach and language syntax highlighting, file tree access, and search makes it a great choice for beginners, according to Friedrich. Indeed, Gitlab had the non-developer in mind when developing the new IDE, Schurter said.

“We’re interested in making this developer experience more accessible to non-developers,” Schurter said. “By having the Web IDE available to everyone, at all times, with no installation or configuration necessary, we hope to make it easier for everyone to contribute.”

The post An Inside Look at What GitLab’s Web IDE Offers Developers appeared first on The New Stack.