Cloud Storage Services for Cloud Native Applications

This article is part of a multipart series on cloud native storage.

Cloud storage is vital to the work of cloud native applications, so choosing from a large portfolio of cloud services can be difficult. Today’s end-to-end, or edge-to-cloud workloads are distributed by design. But the age-old problems of latency, slow transfers of datasets and adequate replication are still with us.

The sheer amount of data is accelerating, driven by cloud based applications, new workloads (AI and ML) and edge-based sensors. IDC has estimated that total worldwide data growing from more than 16 zettabytes (ZB) stored worldwide in 2016 to more than 160ZB stored worldwide by 2025.

Clearly, developers will be looking to Cloud Service Providers (CSPs) for help in storing, accessing and managing those huge amounts of object data. Most of that data is expected to be unstructured data, stored in the hybrid cloud, that will be stored and accessed as cloud based object data. However, enterprise applications are being modified or rewritten for use in hybrid clouds, prompting the need to store block-based and file-based data, along with the object data generated by cloud native applications.

Cloud service providers can ease developers’ tasks with automated cloud storage services, hiding some of the complexity where developers’ actions are not needed. As long as Service Level Agreements (SLAs) are being met by the CSPs, DevOps personnel may not need to get directly involved in hands-on data placement for high availability and operational efficiency.

However, IT organizations should study their options before concluding that all of the needed functionality is baked-in to their CSP’s storage services. Most of the basic functions for cloud-object storage are covered by the major CSPs — Amazon Web Services, Microsoft Azure and Google Cloud Platform (GCP). But DevOps will still need to have a check-off list for enterprise workloads, migrating to the cloud, when examining cloud storage services that will protect mission-critical data. And they will need data-management software, and consoles, to give IT a unified view of all stored data — often provided by systems vendors or software companies (ISVs). Third-party SD-WAN providers are also part of the hybrid cloud storage ecosystem.

Competition among the major CSPs means that cloud storage services, especially for cloud object storage, are generally covering the same ground in terms of cloud stage services functionality. The major CSPs address the need to scale applications in term of capacity, to provide rapid data search, multiple region security and high availability. Also on a CIO’s radar: how will we manage and maintain all of that cloud storage? Performance, scalability, and costs are key concerns for cloud native workloads.

Cloud Storage Services for Cloud Native Workloads

What do cloud-native applications really need? New storage requirements needed for these applications are really a mix of functionality and performance enhancements – although the exact mix will vary based on application specifics. “When you look at storage needs for cloud-native applications, there are broadly two types of new needs that start to emerge,” said Deepak Mohan, Research Director for IDC’s Worldwide Storage Group.

One need is scalability – because cloud-native applications for Internet-scale use cases are typically built on scale-out architectures. This requires the data layer and storage platform to be highly scalable — to scale up to thousands, or hundreds of thousands — of data entities in a flexible manner. “Object storage in the cloud was built with this vision in mind,” Mohan said. ”Large numbers of objects can be stored and delivered in a scalable and reliable manner.”

The other need is performance, Mohan said. “As compute gets more stateless – through transient containers or native serverless options – storage is where application state is being maintained,” he said. “A persistent storage layer, offering a consistent world view to the application, becomes increasingly important. Combined with the distributed nature of many newer applications, storage consistency and latency will be key enablers for such an architecture. With these, applications can have a physically distributed set of compute functionality, with all of them working coherently through persistent ‘state’ in the storage layer.”

CSP Storage Services for Cloud Native Applications

Distributed applications are creating new storage challenges for customers with hybrid clouds or multi-cloud deployments, said Peder Ulander, vice president of product development at Equinix, which links cloud together through its Equinix Cloud Fabric (ECF) service.

“We’re in an interesting phase of enterprise maturity. It’s no longer lift-and-shift, Ulander said. “The biggest challenge is that businesses are moving from data centers to centers-of-data,” he said. “Not everything sits in one place. Data may be in Amazon in Google — or out at the Edge.”

But that creates challenges for cloud native applications that need to tap multicloud data-stores across multiple sites. The large CSPs are all competing to provide services for cloud native, highly distributed applications.

A quick list of cloud storage services includes these from the largest CSPs:

Amazon Web Services (AWS):

AWS is integrating cloud native application support into its Amazon EC2 compute service and its S3 storage service. It joined the Cloud Native Computing Foundation (CNCF) to gain a deeper level of feedback on technologies it must support for cloud native computing and storage.

At the AWS re:Invent 2018 conference, Adrian Cockcroft, vice president of AWS Cloud Architecture Strategy outlined new features of EKS, the Elastic Kubernetes Service, which supports container components for cloud native applications, including cloud storage. A CNCF survey showed that most survey respondents were already using Kubernetes for orchestration of containers with cloud native applications.

Following is a listing of the major AWS storage-related services that can be tapped by developers of cloud native applications:

• AWS S3: Supports durable, scalable and available storage for block, file and object storage. The earliest form of AWS storage, the S3 service provides extensible and highly available data storage, across storage tiers.
• AWS EBS: Elastic Block System supports block storage, which has long been used in enterprise servers and enterprise storage to store application data.
• AWS EFS: Elastic File System supports shared file storage. This is useful when transferring files from servers supporting NFS-based file storage.
• AWS Services for Data Transfer: Supports transfer of large datasets, via Snowball appliances (multiple petabytes per appliance). Another service, the Storage Gateway, connects AWS storage services for purposes of backup/archiving, disaster recovery (DR), cloud data processing, storage tiering and migration.

Microsoft Azure

Microsoft is providing storage services and software tools that address the needs of highly scalable, distributed applications that will be deployed on Microsoft Azure.

Azure supports open-source workloads typical of cloud native applications. The Kubernetes on Azure Container Service (AKS) is aimed at cloud native applications deployed on Azure, providing a Kubernetes-based container solution for software-defined storage. Following is a list of specific Azure storage services, based on storage type, that can be used for cloud native applications:

• Azure Blobs is a massively scalable object store for text and binary data. This is likely to be the most visible Azure service, given the highly scalable nature of cloud native application deployments.
• Azure Files supports managed file shares for cloud and on-premises deployments. Azure Files uses the Server Message Block (SMB) protocol. This service supports enterprise applications that are migrating to the cloud.
• Azure Queues is a messaging store that links application components.
• Azure Tables is a NoSQL store for schema-less storage of structured data.
• Microsoft Azure UltraSSD cloud service, not yet generally available, that provides a tier of low-latency SSD flash storage with high IOPs (I/Os per second).

Google Cloud Platform (GCP) 

Google Cloud Platform has long provided storage for cloud native applications and cloud services. Given Google’s role in the development of Kubernetes orchestration software, and its deep support for AI and ML, cloud native applications are a focus of its cloud storage services, including the following: 

• Google Cloud Storage is a scalable, fully managed, reliable, and cost-efficient object/blob store for storing unstructured data such as images, pictures, and videos. Unstructured data is the fastest growing category for data generated by cloud native applications.
• Google Cloud DataStore is a scalable, fully managed NoSQL document database for web and mobile applications. Cloud native applications often generate data that will be stored in NoSQL databases.
• Google BigTable is a scalable, fully managed NoSQL wide-column database for real-time access and analytics workloads. Real-time computing is a fast-growing segment of the IT market, although real-time applications account for a relatively small slice of the enterprise computing marketplace.

IBM Cloud and Oracle Cloud

For longtime enterprise customers looking to transform aging enterprise workloads, two more cloud storage providers should also be considered: IBM Cloud and Oracle Cloud. These cloud providers, with smaller customer bases than the largest CSPs, emphasize support for enterprise workloads making the transition to cloud, leveraging containers’ software-defined storage.

Both of these CSPs offer services that will run cloud services on-premises at the customer sites. These on-premises cloud services are managed by these two cloud providers (IBM or Oracle), both of which have roots in enterprise computing. Both give customers the option to store data within the original enterprise corporate firewall, for mission-critical data security, for compliance or for data that must be stored within one geographic region.

The Issues at Hand for Cloud Native Applications

 In the age of hybrid cloud, all enterprise applications are being examined with an eye to migrating them to the cloud. “There are refactored applications, and applications that have been picked up and moved,” said John Webster, senior partner and analyst in the Evaluator Group, which focuses on storage market research. “From a storage perspective, those tend to be managed differently.” He said that the major CSPs have competitive storage-service offerings to address customers’ requirements for cloud native applications.

Business goals for cloud migration will make digital transformation, via the cloud, even more important in coming years. Webster estimates that many organizations have targeted moving up to 50 percent of their VMs to the cloud by 2020. That level of cloud migration will take a lot of work by developers, but cloud native momentum is already underway. For many organizations, he said: “It gets put on the board, and those are our marching orders, that’s where we’re going.”

The Cloud Native Computing Foundation is a sponsor of The New Stack.

Feature image by Thana Gu on Unsplash.