The Internet of Things (IoT) and operations technology (OT) spaces are awash in time series data. Sensors churn out time-stamped data at a steady and voluminous pace, and wrangling all that data can be a challenge. But you don’t want to collect a bunch of data and let it just sit there, taking up storage space without providing any value.

So, in the spirit of gaining value from data for IoT/OT, let’s look at how to configure a sensor using InfluxDB and Amazon Web Services (AWS) to collect the type of time-stamped data generated by industrial equipment. You can adapt, expand and scale the concepts here to meet the needs of full industrial production needs. Here’s what we’ll use in the example and why.

• InfluxDB: This time series database just released an updated version that outperforms previous versions in several key areas, such as query performance against high cardinality data, compression and data ingest. Critically for this example, InfluxDB’s cloud products are available in AWS. Having everything in the same place reduces the number of data transfers and data-latency issues in many instances.
• Telegraf: Telegraf is an open source, plugin-based data-collection agent. Lightweight and written in Go, you can deploy Telegraf instances anywhere. With more than 300 available plugins, Telegraf can collect data from any source. You can also write custom plugins to collect data from any source that doesn’t already have a plugin.
• AWS: Amazon Web Services has a whole range of tools and services geared toward IoT. We can tap into some of these services to streamline data processing and analysis.
• M5stickC+: This is a simple IoT device that can detect a range of measurements, including position, acceleration metrics, gyroscope metrics, humidity, temperature and pressure. This device provides multiple data streams, which is similar to what industrial operators face with manufacturing equipment.

InfluxData is the creator of InfluxDB time series platform. InfluxDB empowers developers to build real-time IoT, analytics, and cloud applications with time-stamped data. It is purpose-built to handle the massive volumes of data produced by sensors, systems, or applications that change over time.

Learn More

The latest from InfluxData

InfluxDB and AWS IoT in Action

The following example illustrates one of many possible data pipelines that you can scale to meet the needs of a range of industrial equipment. That may include machines on the factory floor or distributed devices in the field. Let’s start with a quick overview; then we’ll dive into the details.

The basic data flow is:

Device → AWS IoT Core → MQTT (rules/routing) → Kinesis → Telegraf → InfluxDB → Visualization

The M5stickC+ device is set up to authenticate directly with AWS. The data comes into AWS IoT Core, which is an AWS service that uses MQTT to add the data to a topic that’s specific to the device. Then there’s a rule that selects any data that goes through the topic and redirects it to the monitoring tool, Amazon Kinesis.

A virtual machine running in AWS also runs two Docker containers. One contains an instance of Telegraf and the other an instance of InfluxDB. Telegraf collects the data stream from Kinesis and writes it to InfluxDB. Telegraf also uses a DynamoDB table as a checkpoint in this setup so if the container or the instance goes down, it will restart at the correct point in the data stream when the application is up again.

Once the data is in InfluxDB, we can use it to create visualizations.

OK, so that’s the basic data pipeline. Now, how do we make it work?

Device Firmware

The first step is creating a data connection from the M5 device to AWS. To accomplish this we use UI Flow, a drag-and-drop editor that’s part of the M5 stack. Check out the blocks in the image below to get a sense for what the device is collecting and how these things map to our eventual output.

We can see here that this data publishes to the MQTT topic m5sticks/MyThingGG/sensors.

AWS IoT Core Rules

With the device data publishing to an MQTT broker, next we need to subscribe to that topic in AWS IoT Core. In the topic filter field, enter m5sticks/+/sensors to ensure that all the data from the device ends up in the MQTT topic.

Next, we need to create another rule to ensure that the data in the MQTT topic goes to Kinesis. In IoT Core, you can use a SQL query to accomplish this:

SELECT *, topic(2) as thing, 'sensors' as measurement, timestamp() as timestamp FROM 'm5sticks/+/sensors'

In an industrial setting, each device should have a unique name. So, to scale this data pipeline to accommodate multiple devices, we use the +` wildcard in the MQTT topic to ensure that all the data from all devices end up in the correct place.

This query adds a timestamp to the data so that it adheres to line protocol, InfluxDB’s data model.

Telegraf Policy

Now that data is flowing from the device to Kinesis, we need to get it into InfluxDB. To do that, we use Telegraf. The code below is the policy that determines how Telegraf interacts with Kinesis. It enables Telegraf to read from the Kinesis stream and enables read and write access to DynamoDB for checkpointing.

{
	"Version": "2012-10-17"
	"Statement": [
		{
			"Sid": "AllowReadFromKinesis",
			"Effect": "Allow",
			"Action": [
				"kinesis:GetShardIterator",
				"kinesis:GetRecords",
				"kinesis:DescribeStream"
			],
			"Resource": [
				"arn:aws:kinesis:eu-west-3:xxxxxxxxx:stream/InfluxDBStream"
			]
		},
		{
			"Sid": "AllowReadAndWriteDynamoDB",
			"Effect": "Allow",
			"Action": [
				"dynamodb:PutItem",
				"dynamodb:GetItem"
			],
			"Resource": [
				"arn:aws:kinesis:eu-west-3:xxxxxxxxx:table/influx-db-telegraf"
			]		
		}
	]
}

Telegraf Config

The following Telegraf config uses Docker container networking, the Telegraf Kinesis Consumer plugin to read the data from Kinesis and the InfluxDB v2 output plugin to write that data to InfluxDB. Note that the string fields match the values from the device firmware UI.

[agent]
debug = true

[[outputs.influxdb_v2]]

## The URLs of the InfluxDB Cluster nodes.
##
## Multiple URLs can be specified for a single cluster, only ONE of the
## urls will be written to each interval.
## urls exp: http://127.0.0.1:8086
urls = ["http://influxdb:8086"]

## Token for authentication.
token = "toto-token"

## Organization is the name of the organization you wish to write to; must exist.
organization = "aws"

## Destination bucket to write into.
bucket = "toto-bucket"


[[inputs.kinesis_consumer]]
## Amazon REGION of kinesis endpoint.
region = "eu-west-3"

## Amazon Credentials
## Credentials are loaded in the following order
## 1) Web identity provider credentials via STS if role_arn and web_identity_token_file are specified
## 2) Assumed credentials via STS if role_arn is specified
## 3) explicit credentials from 'access_key' and 'secret_key'
## 4) shared profile from 'profile'
## 5) environment variables
## 6) shared credentials file
## 7) EC2 Instance Profile

## Endpoint to make request against, the correct endpoint is automatically
## determined and this option should only be set if you wish to override the
## default.
##   ex: endpoint_url = "http://localhost:8000"
# endpoint_url = ""

## Kinesis StreamName must exist prior to starting telegraf.
streamname = "InfluxDBStream"

## Shard iterator type (only 'TRIM_HORIZON' and 'LATEST' currently supported)
# shard_iterator_type = "TRIM_HORIZON"

## Max undelivered messages
## This plugin uses tracking metrics, which ensure messages are read to
## outputs before acknowledging them to the original broker to ensure data
## is not lost. This option sets the maximum messages to read from the
## broker that have not been written by an output.
##
## This value needs to be picked with awareness of the agent's
## metric_batch_size value as well. Setting max undelivered messages too high
## can result in a constant stream of data batches to the output. While
## setting it too low may never flush the broker's messages.
# max_undelivered_messages = 1000

## Data format to consume.
## Each data format has its own unique set of configuration options, read
## more about them here:
## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
data_format = "json"

## Tag keys is an array of keys that should be added as tags.
tag_keys = [
	"thing"
]

## String fields is an array of keys that should be added as string fields.
json_string_fields = [
	"pressure",
	"xGyr",
	"yAcc",
	"batteryPower",
	"xAcc",
	"temperature",
	"zAcc",
	"zGyr",
	"y",
	"x",
	"yGry",
	"humidity"

]

## Name key is the key used as the measurement name.
json_name_key = "measurement"

## Time key is the key containing the time that should be used to create the
## metric.
json_time_key "timestamp"

## Time format is the time layout that should be used to interpret the
## json_time_key. The time must be 'unix', 'unix_ms' or a time in the 
## "reference_time".
##   ex: json_time_format = "Mon Jan 2 15:04:05 -0700 MST 2006"
##       json_time_format = "2006-01-02T15:04:05Z07:00"
##       json_time_format = "unix"
##       json_time_format = "unix_ms"
json_time_format = "unix_ms"

## Optional
## Configuration for a dynamodb checkpoint
[inputs.kinesis_consumer.checkpoint_dynamodb]
## unique name for this consumer
app_name = "default"
table_name = "influx-db-telegraf"

Docker Compose

This Docker compose file gets uploaded to an EC2 instance using SSH.

version: '3'

services:
  influxdb:
    image: influxdb:2.0.6
    volumes:
      # Mount for influxdb data directory and configuration
      - influxdbv2:/root/.influxdbv2
    ports:
      - "8086:8086"
# Use the influx cli to set up an influxdb instance
  influxdb_cli:
    links:
      - influxdb
    image: influxdb:2.0.6
# Use these same configurations parameters in you telegraf configuration, mytelegraf.conf
	# Wait for the influxd service in the influxdb has been fully bootstrapped before trying to set up an influxdb
   restart: on-failure:10
   depends_on:
   	- influxdb
  telegraf:
    image: telegraf:1.25-alpine
    links:
      - influxdb
    volumes:
      # Mount for telegraf config
      - ./telegraf.conf:/etc/telegraf/telegraf.conf
    depends_on:
      - influxdb_cli

volumes:
  influxdbv2:

Visualization

Once your data is in InfluxDB, users running InfluxDB can create visualizations and dashboards using your tool of choice. InfluxDB offers a native integration with Grafana and supports SQL queries using Flight SQL-compatible tools.

Conclusion

Building reliable data pipelines is a critical aspect of industrial operations. This data provides key information about the current state of machinery and equipment, and can train predictive models used to improve machinery operations and effectiveness. Combining leading technologies like InfluxDB and AWS provides the tools necessary to capture, store, analyze and visualize this critical process data.

The use case described here is just one way to build a data pipeline. You can update, scale and amend it to accommodate a wide range of industrial IoT/OT operations or build something completely custom using the range of IoT solutions available for InfluxDB and AWS.

The post Creating an IoT Data Pipeline Using InfluxDB and AWS appeared first on The New Stack.

At its annual Dell Technologies World conference in Las Vegas last week, Dell joined a spate of companies this year in releasing new products that address emerging needs in edge management, secure generative AI development, zero trust security and multicloud workload management. It’s no surprise that these are the hottest sectors of enterprise IT products and services sales here in 2023.

New items Dell introduced include the NativeEdge platform; a partnership between itself and Nvidia to build Project Helix, a secure on-premises-based generative AI management package; Project Fort Zero, another new crack at providing an end-to-end zero trust security solution; and updates for its multicloud app-service management platform, Apex.

Dell NativeEdge

Dell had only a loosely coordinated edge strategy for a few years until last October when the company previewed a new development called Project Frontier, which was to result in a standalone edge operations software and hardware platform sometime in 2023.

“Sometime in 2023” is now here. Project Frontier is now closed, and Dell said its replacement, dubbed NativeEdge, is the industry’s “only” edge operations software platform delivering secure device onboarding at scale, remote management and multicloud application orchestration. Companies such as IBM, AWS, Google, HP, Microsoft, Hitachi, Oracle and several others may dispute the “only” claim, however. Here is Gartner’s most recent edge computing market report.

“This is the industry’s only edge operation platform that mixes up and delivers three things in parallel: secure device onboarding at scale; remote management of devices in those locations; and multicloud application orchestration to edge devices from data centers to make sure that the edge outcome end-to-end is deployed in the lifecycle,” Dell SVP of Edge Solutions Gil Schneorson told reporters.

Edge operations, which make use of automated processes, are generally found in locations ranging from manufacturing floors and retail stores to remote wind turbines and long-distance hospital operating rooms.

Use cases for edge computing are growing because more organizations want to manage and secure data at the source but often have limited IT support to do it.

For example, a large manufacturer may need to automate packaging and shipping across its numerous factory sites in various geographies. This means connecting multiple technologies, such as IoT, streaming data and machine vision, which requires dedicated devices to run multiple software applications across locations. Testing and deploying infrastructure to run the applications can take months. With NativeEdge, the manufacturer can consolidate its technology stacks using existing investments and reduce the time to deploy edge assets and applications from months to weeks, Schneorson said. The platform uses automation to streamline edge operations and helps the manufacturer securely roll out new applications to all sites from a central location.

NativeEdge is designed to run any enterprise edge use case with automated deployment, Schneorson said, integrating with various Dell data center hardware nodes. NativeEdge includes built-in zero trust internal use-confidential capabilities, Schneorson said.

The NativeEdge software platform will be available in 50 countries beginning in August, the company said.

Project Helix Partnership with Nvidia

Everybody needs “easier” when it comes to incorporating generative AI into an enterprise system, no matter where its use is being considered. Pre-built AI data models are one major step in that direction.

Dell and Nvidia revealed that Project Helix will result in a series of full-stack solutions with technical expertise and prebuilt tools based on their own infrastructure and software. This includes a complete blueprint to help enterprises use their proprietary data and more easily deploy generative AI responsibly and accurately, Dell said.

“Project Helix gives enterprises purpose-built AI models to more quickly and securely gain value from the immense amounts of data underused today,” Jeff Clarke, Dell Technologies’ vice chairman and co-chief operating officer, said in a media advisory. “With highly scalable and efficient infrastructure, enterprises can create a new wave of generative AI solutions that can reinvent their industries.”

Blueprints for on-premises generative AI Project Helix will support the complete generative AI lifecycle, from infrastructure provisioning, modeling, training, fine-tuning, application development and deployment, to deploying inference and streamlining results, Dell said. The validated designs will use Dell PowerEdge servers, such as the PowerEdge XE9680 and PowerEdge R760xa, which are optimized to deliver performance for generative AI training and AI inferencing, the company said. They will combine with Nvidia H100 Tensor Core GPUs and networking to form the infrastructure backbone for these workloads.

Dell did not indicate whether the Helix models could be run optimally on other brands of servers. Dell Validated Designs based on the Project Helix initiative will be available through traditional channels and Apex flexible consumption options beginning in July.

Dell Apex Multicloud Management

Dell Apex software, introduced in May 2021, is designed to work across all cloud platforms, public cloud storage software, client devices and computing environments. New additions to the company’s multicloud portfolio span data centers to public cloud and client devices include:

Dell, Microsoft, VMware and Red Hat platforms for specific cloud-based use cases:

Dell Apex Cloud Platform for Microsoft Azure features full-stack software integration and automated lifecycle management through Microsoft native management tools. The platform is designed for application modernization and is said to offer faster time to value of Azure based on Azure Arc-enabled infrastructure with consistent operations and governance across on-premises data centers, edge locations, and the Azure public cloud using Azure Arc.

Apex Cloud Platform for Red Hat OpenShift aims to simplify container-based application development and management, wherever applications are developed and deployed, through full stack software integration and automation with Kubernetes. Users can run containers and virtual machines side by side, natively within Kubernetes, for a wide variety of workloads — including AI/ML and analytics, with broad GPU support across any hybrid cloud footprint, Dell said.

Apex Cloud Platform for VMware gives users the flexibility to deploy vSphere on a fully integrated system with Dell software-defined storage. It joins Dell Apex Private Cloud and Dell Apex Hybrid Cloud within the broader Dell Apex portfolio to offer more choices for VMware users.

Zero Trust Security News

This is still on the drawing and testing boards for the next year until further notice. Project Fort Zero is designed to make zero trust more available for mid-range and smaller enterprises and is engineered to provide a hardened end-to-end zero trust security solution for global organizations to protect against cyberattacks.

Fort Zero is part of an ecosystem of more than 30 technology companies and will result in a validated, advanced maturity zero trust solution within the next 12 months, Dell said. “Zero trust is designed for decentralized environments, but integrating it across hundreds of point products from dozens of vendors is complex — making it out of reach for most organizations,” Herb Kelsey, industry CTO of government projects at Dell, told reporters. “We’re helping global organizations solve today’s security challenges by easing integration and accelerating adoption of Zero Trust.”

The fully configured Fort Zero solution will serve a variety of use cases, including on-premises data centers; remote or regional locations such as retail stores where secure, real-time analysis of customer data can deliver a competitive advantage; in the field, where a temporary implementation is needed in places with intermittent connectivity, such as airplanes or vehicles, for operational continuity.

The post Dell Intros New Edge, Generative AI, Cloud, Zero Trust Prods appeared first on The New Stack.

With around 500,000 residents in town, and over a million in its metropolitan area, Gothenburg, Sweden has plenty of people and ground to support. Like any city, there are services that need to be offered, and systems that need to be maintained over time. And it’s always easier to instrument things and monitor them from somewhere else than it is to drive out and check on pipes, lamps and streetlights in person.

So the basic appeal of the Internet of Things (IoT) on the city level is to save time and gain efficiency by eliminating travel time and labor. That’s certainly what appealed to Hans Wendt, the project manager for the Gothenburg City Information Platform and IoT project. Wendt likes saving resources, and the city’s IoT project even extends to saving water: If it rains, sensors tell city sprinkler systems to pause watering activities for some time period.

But building out this system of sensors across multiple aspects of the city’s services wasn’t easy.

“The market is building different siloed solutions for each company, making it expensive to manage. By building a common solution to manage the issue, [we] make it much easier,” he said.

To build a solution to handle multiple IoT devices, platforms and problems, Gothenburg chose to build on top of Red Hat OpenShift. But first, it needed to build a proof of concept. And for that, it needed something to monitor and measure with sensors.

Wendt and Frederik Lehtonen, product owner for OpenShift at Gothenburg, chose bath water. After a proof of concept measuring bath water temperature coming out of people’s homes, they looked at other watering projects. And as a result, they’ve already cut water usage for managing 50,000 trees by 75%. They did this by connecting newly planted trees to sensors to monitor moisture levels and adjust watering.

Red Hat OpenShift is for innovation without limitation. Bring big ideas to life with the hybrid cloud platform open to any app, team, or infrastructure.

Learn More

The latest from Red Hat

Why Water?

The project started two years ago with the goal of building a platform for all of the city of Gothenburg IT. The team wanted to provide a simple way for the city’s IT teams to build projects more efficiently. While the goal was clear, the implementation quickly forced the city’s workers to collaborate across departments.

That meant IT collaborating with the City Parks Department. The first test was run inside the city’s public green areas and sports centers. As part of the innovation project “Sustainable Smart Parks,” the project has expanded and come to include multiple city departments, a university and a national agency.

Using temperature and humidity sensors combined with satellite-gathered information, the city can manage its park watering system automatically. Using water meters, the city can detect leaks remotely, which is a big deal because 25% of the water produced for Sweden is lost to leakage each year.

Until now, meter readers had to physically check a card on the pipe itself to measure outflows. Now the city can measure it automatically every hour via IoT sensors. This eliminates not only the physical work and delay associated with meter reading, but also eliminates human error from the process, Wendt said.

Open by Default

The project is operating under a city leadership mandate to digitize as much as possible by 2025 and to use open systems for its infrastructure. This is one of the reasons the city chose Red Hat OpenShift, but it’s also behind another decision. The team decided to cut through the IoT noise by building their platform with FIWARE.

FIWARE is a curated framework of open source software platform components. At the center is the FIWARE Context Broker, which brings management to contextual data around information streams. Specifically, this enables things like IoT data streams, monetization of those streams and processing of the data.

Building on top of those two open systems enables Wendt and his team to comply with the security and data retention needs that come with running civic infrastructure. And that’s an important building block for a city that has three new IoT projects in the works and over 50 in the planning stages. That platform will soon be handling far more water data and will also take up streetlight management.

Under the city mandate, Wendt and his team are also working with IT to enable a data lake model for the information pouring in from sensors. They are embracing the self-service data model and hope to provide a high-quality data analysis platform to city workers.

Behind all these plans is a central desire to help all of Sweden, not just Gothenburg, Wendt said. Open source enables sharing.

Using what others have made and creating possibilities for working together in the future allows money being spent on this work to be saved elsewhere if other municipalities can also use these platforms. Plus, there’s the inherent cost savings of open source itself, said Wendt. “Fiware was a cheaper solution and easy to test and use also. [That was also] a benefit.”

For the future, Wendt sees some hurdles that still require clearing in the marketplace. He said that scale is the biggest challenge for the future, and more specifically, the challenge of asset management as sensors blossom from hundreds to hundred thousands in the field. Managing that many devices presents a challenge for the platform they are building, but also an opportunity to solve a problem others are having.

But the devices themselves remain a source of interest and innovation. Wendt said the team has plans for all manner of devices depending on battery power, frequency of communication, size and durability. They are currently considering placing sensors in waste bins to detect when they need to be emptied.

Still, management could be an issue for data, physical maintenance and lines of control for each device. Who has to go out and change a dead sensor in the lake? Parks? IT? The local scuba club?

Despite these challenges, Wendt is confident Gothenburg has built a platform that can meet those needs. Building on Kubernetes, he said, allows for each workload to be securely isolated, and the data associated with it can be encrypted in transit and at rest, to keep it in compliance with regulations like GDPR. That will also be important if and when the team rolls its platform out to the city healthcare systems.

The post Gothenburg, Sweden Used Open Source IoT to Drastically Cut Water Waste appeared first on The New Stack.

Creating an Internet of Things (IoT) app to monitor a house plant is a pragmatic starting place to learn about data that changes over time. It’s useful for anyone who loves the idea of indoor gardening but forgets to check their plants regularly.

This project is accessible to everyone from students working on a science fair project to botanists monitoring exotic plant nurseries. There are many ways to monitor a houseplant, but this is the high-tech way — with sensors and advanced software systems.

For this project, we’ll use InfluxDB, a time series platform that specializes in storing sequential data as it appears over time. This is useful when comparing data, creating alerts for specific thresholds and monitoring events in the physical and virtual worlds alike.

The full list of supplies, a schematic drawing for your breadboard (with microcontroller of choice) and the source code are all available from my git repository if you want to follow along for your own edification or desperately need to keep a houseplant alive.

The Architecture

An IoT sensor tracks my plant’s health metrics at timed intervals. We classify the data it collects as time series data because it includes a timestamp, which appears as the first column in storage. The IoT sensors generate data about the plant’s health. Then I use Telegraf, an open source collection agent, and the Python client library to collect that data and send it to the storage layer, InfluxDB. The Plotly graphing library provides the data visualization. I coded the project in Python and it uses the Flask library for routing.

InfluxData is the creator of InfluxDB time series platform. InfluxDB empowers developers to build real-time IoT, analytics, and cloud applications with time-stamped data. It is purpose-built to handle the massive volumes of data produced by sensors, systems, or applications that change over time.

Learn More

The latest from InfluxData

Getting Started

Instruments:

• A plant to monitor
• A Particle Boron or similar microcontroller
• At least one IoT sensor for your plant
• A breadboard
• Jump wires

I use four sensors to generate the following five data points:

• Air temperature
• Humidity
• Light
• Soil temperature
• Soil moisture

Microcontroller

I use the Boron microcontroller and set the device up through the company website. To receive data from the microcontroller itself, I connect it to my laptop via a USB cable. Microcontroller setup depends on which microcontroller you selected for use. Your microcontroller might provide other connection options, including Bluetooth or small server access over TCP/IP.

Follow the instructions provided by your microcontroller’s manufacturer until you receive input from your sensors. You don’t need to make sense of the data yet; just make sure your microcontroller is sending raw data.

InfluxDB

Sign into InfluxDB. Create a bucket, which is where InfluxDB stores data. We will connect to InfluxDB via an API. The next step is to create the required credentials and tokens.

Code

Writing data into InfluxDB is straightforward and starts with the client library. I use InfluxDB’s Python client library for this project. The code below is an example of how you can write code to send raw data from your microcontroller’s sensors to InfluxDB.

def write_to_influx(self,data):
	p = (influxdb_client.Point("sensor_data")
                    	.tag("user",data["user"])
                    	.tag("device_id",data["device"])                     
                    	.field(data["sensor_name"], int(data["value"])
                    	))
	self.write_api.write(bucket=self.cloud_bucket, org=self.cloud_org, record=p)
	print(p, flush=True)

Tags

Measurements are the InfluxDB equivalent to tables in a relational database. My code makes good use of tags. Tags aren’t required but come in handy because they’re metadata that makes the data easier to understand and work with. In this case, they can specify different plants, devices or something else depending on how complicated you want to get.

Querying Data

Queries return tables similar to the one below.

Before I can query my data, I need to initialize the Flight SQL client.

from flightsql import FlightSQLClient

Followed by:

# This is our flight client setup, it’s how we will query from IOX
    	# we need to remove the Https:// from our host
    	host = host.split("://")[1]
    	self.flight_client = FlightSQLClient(host=host,
                     	token=token,
                     	metadata= {'bucket-name': bucket}
        	             )
                        
    	self.cloud_bucket = bucket
    	self.cloud_org = org

Below is a basic SQL query to retrieve data from InfluxDB.

SELECT {sensor_name}, time FROM sensor_data WHERE time > (NOW() - INTERVAL '2 HOURS') AND device_id='{deviceID}'

Before we can retrieve and read the data, we have to convert it to Pyarrow format. The code below is a function that includes the query and connection to Flight SQL to retrieve the data.

def querydata(self, sensor_name, deviceID) -> DataFrame:   	
 
        query = self.flight_client.execute(f"SELECT {sensor_name}, time FROM sensor_data WHERE time > (NOW() - INTERVAL '2 HOURS') AND device_id='{deviceID}'")
    	
        # Create reader to consume result
    	reader = self.flight_client.do_get(query.endpoints[0].ticket)
 
    	# Read all data into a pyarrow.Table
    	Table = reader.read_all()
    	print(Table)
 
   	# Convert to Pandas DataFrame
    	df = Table.to_pandas()
    	df = df.sort_values(by="time")
    	print(df)
    	return df

You can call the previous query and substitute the variables for your selections, including bucket, sensor and device. The returned result allows you to graph your incoming data. The return df method pulls our data out in a data frame format.

Data Frames

Pandas DataFrames are two-dimensional data structures that enable fast data analysis and processing. We convert our data to a DataFrame to make it easier to work with in Python. There are a few other data output options to choose from if you prefer a different style.

@app.callback(Output("store", "data"), [Input("button", "n_clicks")])
def generate_graphs(n):
# Generate graphs based upon pandas data frame. 
    df = influx.querydata( "soil_temperature", graph_default["deviceID"] )
    soil_temp_graph = px.line(df, x="time", y="soil_temperature", title="Soil Temperature")
 
    df = influx.querydata( "air_temperature", graph_default["deviceID"] )
    air_temp_graph= px.line(df, x="time", y="air_temperature", title="Air Temperature")
 
    df = influx.querydata( "humidity", graph_default["deviceID"] )
    humidity_graph= px.line(df, x="time", y="humidity", title="humidity")
 
    df = influx.querydata( "soil_moisture", graph_default["deviceID"] )
    soil_moisture= px.line(df, x="time", y="soil_moisture", title="Soil Moisture")
 
    df = influx.querydata( "light", graph_default["deviceID"] )
    light_graph= px.line(df, x="time", y="light", title="light")

The graphing library expects you to return a data frame for visualization. This is the end result of querying for the data points. The images below are hard-coded graphs that illustrate the data points. Different tabs display different graphs and track separate metrics. This is just a small portion of the project’s capabilities.

Conclusion

Check out my presentation centered around Plant Buddy for a more in-depth discussion on this project and the InfluxDB ecosystem at large. Our community page has other great examples of exciting projects. Now get started with InfluxDB and build something cool!

The post Building a Plant Monitoring Tool with IoT appeared first on The New Stack.

Choosing the right database is essential for any organization that wants to efficiently manage and analyze its time series data. The right database will be able to handle the volume and complexity of the data being generated, integrate with existing systems seamlessly, and be cost-effective.

However, selecting the wrong database can result in performance issues, data loss and a significant waste of time and resources. Therefore, it is crucial to carefully evaluate and choose a database that is best suited for the organization’s specific needs, considering factors such as data volume, query complexity, integration and cost.

Here’s how to evaluate your choices, and what to consider — along with some best practices for modeling time series data.

Evaluating a Time Series Database

Choosing the right time series database for your use case can be a daunting task, as there are many options available with varying features and capabilities. Here are some factors to consider.

Data volume and velocity. Consider the expected volume of time series data that you will be collecting and storing. Choose a database that can handle the expected data volume, and that can scale as those volumes increase over time.

Query complexity. Consider the types of queries that you will be running. Some databases are better suited for simple queries, while others offer more advanced query languages and functions for complex analytics. Choose a database that can handle the complexity of your queries, and that offers a query language that is well-suited for your use case.

Integration with existing systems. Consider the systems that you already have in place, such as monitoring and analytics tools, and choose a database that can integrate seamlessly with those systems. This will make it easier to manage and analyze your time series data.

Security. Choose a database that offers robust security features, such as encryption and access control, and meets your data’s security requirements.

Cost and licensing. Consider the database’s cost, as some features and capabilities may bring a higher price tag. Also think about long-term costs, including licensing fees, maintenance costs and scalability.

Support and community. Finally, consider the support and community around the time series database. Look for databases with active development and a strong community of users who can provide support and share best practices.

Kinetica is available as a fully managed service in the Cloud. Continued focus on core features and an improved Workbench interface drives growth among enterprises, new tech and government agencies including some of the largest IoT data analytics implementations in the world.

Learn More

The latest from Kinetica

Best Practices for Modeling Time Series Data

To make sure your data set is useful for analysis and decision-making, follow these best practices:

Define the Business Problem

Before collecting any data, it’s important to clearly define the business problem you are trying to solve. What do you want to learn from this data? This will help you determine the appropriate granularity and time interval you’ll need to run queries and do analysis.

Choose the Granularity and Time Interval

Choosing the right granularity and time interval is crucial for accurate analysis, efficient storage and efficient query processing. It involves finding the right balance between capturing enough detail to support your analysis and keeping storage space requirements and computational costs manageable.

Granularity refers to the level of detail captured by each data point in a time series, while time interval determines how often data points are recorded.

One of the key tradeoffs when selecting granularity is the balance between detail and storage space. High granularity means capturing more detail in each data point, which provides more information for analysis but also takes up more storage space. On the other hand, low granularity captures less detail but requires less storage space.

The choice of granularity and time interval also depends on the type of data being collected and the specific use case. For instance, in applications like sensor data collection, where data is generated continuously, a higher granularity is often necessary to capture short-term changes accurately. In contrast, for data generated periodically, such as sales data, lower granularity is often sufficient.

Selecting the time interval also involves tradeoffs between data accuracy and storage space. A shorter time interval provides more data points but also increases storage requirements, whereas a longer time interval reduces the storage requirements but may miss important events that occur between measurements.

In addition to these tradeoffs, consider the downstream analysis that will be performed on the data. For example, if your analysis requires calculating daily averages, a time interval of one hour may provide too much data and lead to unnecessary computational costs.

Plan for Missing or Incomplete Data

Missing data is a common issue in time series data. Make a plan for how to handle it, whether it’s through imputation, interpolation or simply ignoring the missing data.

Normalize Your Data

Time-series data often comes from multiple sources with different units of measurement, varying levels of precision and different data ranges. To make the data useful for analysis, it’s essential to normalize it.

Normalization is the process of transforming data into a common scale to facilitate accurate comparison and analysis. In the next section, we’ll explore the importance of normalization for accurate comparisons and analytics, and methods for normalizing data.

Why Normalization of Time Series Data Matters

Normalization is essential for accurate data comparisons and analysis. If the data is not normalized, it may lead to biased results or incorrect conclusions.

For example, imagine comparing the daily average temperature in two cities. If one city reports the temperature in Celsius, and the other city reports it in Fahrenheit, comparing the raw data without normalizing it would produce misleading results. Normalizing the data to a common unit of measurement, such as Kelvin, would provide an accurate basis for comparison.

Normalization is also crucial for analytics, such as machine learning and deep learning. Most algorithms are sensitive to the magnitude of data, and normalization helps ensure that the algorithm weights each input feature equally. Without normalization, some features with large ranges may dominate the model’s training process, leading to inaccurate results.

Methods for Normalizing Data

There are several methods for normalizing data, including:

Min-max scaling. This method scales the data to a specific range, usually between 0 and 1 or -1 and 1. It’s calculated by subtracting the minimum value of the data and then dividing it by the range between the maximum and minimum values.

Z-score normalization. This method scales the data to have a mean of 0 and a standard deviation of 1. It’s calculated by subtracting the mean of the data and then dividing it by the standard deviation.

Decimal scaling. This involves scaling the data by moving the decimal point of each value to the left or right. The number of decimal places moved depends on the maximum absolute value of the data.

Log transformation. This involves transforming the data using a logarithmic function. It’s often used for data with a wide range of values.

Choosing an Appropriate Data Type

Different types of data can be stored in different formats, and choosing the wrong data type can lead to inefficient storage and retrieval, as well as inaccurate analysis.

The first consideration is to choose a data type that accurately represents the data being collected. For example, if you are collecting temperature data, you may choose a data type of float or double precision, depending on the precision required. If you are collecting binary data, such as on/off states, you may choose a Boolean data type.

Another consideration is the size of the data. Using the smallest data type that accurately represents the data can help to reduce storage requirements and improve query performance.

For example, if you are collecting integer values that range from 0 to 255, you can use an unsigned byte data type, which only requires one byte of storage per value. This is much more efficient than using a larger data type such as an integer, which requires four bytes of storage per value.

Consider the size of the database and the volume of data being collected. For example, if you are collecting high-frequency data, such as millisecond-level sensor data, you may want to use a data type that supports efficient compression, such as a delta-encoded data type. This can help to reduce storage requirements while maintaining data fidelity.

Finally, consider the types of operations that will be performed on the data. For example, if you will be performing aggregate operations such as summing or averaging over large data sets, you may want to use a data type that supports efficient aggregation, such as a fixed-point decimal data type.

How to Detect and Handle Outliers

Outliers are data points that deviate significantly from the expected values in a time series. They can occur due to various reasons such as measurement errors, equipment malfunctions, or actual anomalies in the data. Outliers can have a significant impact on the analysis of time series data, as they can skew results and affect the accuracy of models.

Detecting and handling outliers in time series data is an important task to ensure accurate analysis and modeling. Here are some methods for doing so.

Visual inspection. One of the simplest methods for detecting outliers is simply by looking for them in your data. Look for data points that are significantly different from the rest of the data.

Statistical methods. Various statistical methods can be used to detect outliers in time series data, such as the z-score method, which identifies data points that are more than a certain number of standard deviations away from the mean.

Machine learning algorithms. Algorithms can be trained to detect outliers in time series data. For example, an autoencoder neural network can be trained to reconstruct the time series data and identify data points that cannot be reconstructed accurately.

Once outliers are identified, there are several methods for handling them, including:

Removing outliers. The simplest method for handling outliers is to remove them from the data set. However, this approach can result in the loss of valuable information and should be used judiciously.

Imputing values. Another approach is to impute the values of the outliers based on the surrounding data points. This method can be useful when the outliers are caused by measurement errors or other minor anomalies.

Model-based approaches. These can be used to handle outliers by incorporating them into the model. For example, robust regression models can be used that are less sensitive to outliers.

Best Practices for Managing Time Series Databases

Maintaining a time series database requires careful planning and execution to ensure its smooth operation and optimal performance. Here are some best practices for managing and maintaining these databases, covering four essential areas: monitoring and troubleshooting, backup and disaster recovery, scaling and performance tuning, and security and access control.

Monitoring and Troubleshooting

To ensure the optimal performance of a time series database, it must be monitored continuously. Monitoring can help detect issues such as slow queries, high CPU utilization and network latency. Various tools can be used to monitor time series databases, including monitoring dashboards, alerts, and logging. These tools can help identify and isolate issues quickly and efficiently, reducing downtime and ensuring business continuity.

Methods for monitoring and troubleshooting time series databases are essential for ensuring the availability and performance of the database.

Some common methods for monitoring and troubleshooting time series databases include:

Monitoring tools. There are several monitoring tools available for time series databases, such as Prometheus, InfluxDB, and Grafana. These tools can provide real-time metrics and alerts on the performance of the database, such as query response time, CPU and memory usage, and disk space utilization.

Logging. A critical tool for troubleshooting issues in time series databases,  logging can help identify errors, warnings and other issues that may be affecting the performance of a database. It’s important to configure logging to capture the necessary information for diagnosing issues, such as query logs and server logs.

Performance tuning. Performance tuning is the process of optimizing the database for maximum performance. This can involve adjusting the configuration settings of the database, such as buffer sizes, thread pools and cache sizes, to ensure that the database is running efficiently.

Load testing. This involves simulating a high volume of traffic on the database to test its performance under stress. This can help identify any bottlenecks or performance issues that may be affecting the database.

Health checks. Regular health checks can help detect and prevent issues with the database. These checks can include verifying the database’s disk space, checking for hardware failures, and testing the database’s failover and recovery mechanisms.

Backup and Disaster Recovery

To protect the data against loss or correction, it’s crucial to back up your time series database. Disaster recovery plans should be in place to ensure that the business can recover from any unforeseen events. The frequency with which you should run backups should be determined by the value of your data and the potential impact of its loss.

Here are some methods for backup and disaster recovery:

Regular backups. One of the most common and effective methods for backing up time series data is to take regular backups at specified intervals, and to perform them for different levels, such as the file system, database, or application.

Replication. Replication involves creating a copy of your data and storing it on a different system. This can be a good strategy for disaster recovery as it provides a redundant copy of your data that can be quickly accessed in case of a system failure. Replication can be synchronous or asynchronous; consider the cost and complexity of replication when deciding on a backup strategy.

Cloud-based backups. Many time series databases offer cloud-based backup solutions that can automatically back up your data to remote servers. This can be a good option for organizations that want to ensure their data is protected without having to manage backup infrastructure themselves.

Disaster recovery plans. In addition to backups, having a comprehensive disaster recovery plan in place is essential for quickly restoring data in case of unexpected events. A disaster recovery plan should include procedures for restoring backups, identifying critical systems and data, and testing recovery procedures regularly.

Monitoring and testing. Regularly monitor and test backup and disaster recovery procedures to ensure that they are working effectively. Regular testing can identify potential issues and help optimize backup and recovery processes.

Scaling and Performance Tuning

Time series databases can grow significantly over time, and as such, they require scaling to ensure optimal performance. Scaling can be achieved horizontally or vertically, and the choice of method depends on the specific use case. Performance tuning involves optimizing the database’s configuration and queries to ensure that queries are processed efficiently and quickly.

To ensure that a time series database can handle these increasing demands, here are several methods to help scale the database and fine-tune its performance:

Partitioning. Partitioning is a technique for dividing data into smaller subsets, making it easier to manage and query. By partitioning data, queries can be executed more efficiently, and it can reduce the amount of data that needs to be scanned, resulting in faster query response times.

Indexing. Indexing helps optimize query performance by creating indexes on columns that are frequently used in queries. This allows the database to quickly find relevant data and return results faster.

Caching. Caching is a technique for storing frequently accessed data in memory to reduce the need for repeated access to disk storage. This can significantly improve query response times and overall database performance.

Compression. This method reduces the storage space required for data by encoding it in a more compact format. By compressing data, storage costs can be reduced, and query response times can be improved as less data needs to be read from the disk.

Sharding. Sharding is a technique for horizontally partitioning data across multiple servers, allowing for greater scalability and higher throughput. This method is particularly useful for databases that are experiencing rapid data growth and requires more processing power to handle queries.

Hardware upgrades. Upgrading hardware, such as increasing the amount of RAM or adding more CPU cores, can significantly improve the performance of a time series database. By increasing the processing power and memory available, queries can be executed faster, and the database can handle more concurrent requests.

Security and Access Control

Time series data can be sensitive and valuable, and as such, it requires robust security measures. Access control is crucial to controlling who can access the data and what actions they can perform.

Here are some methods for securing time series databases, to protect sensitive data and prevent unauthorized access.

Role-based access control (RBAC). RBAC, a widely used approach for controlling access to databases, assigns specific roles to users, which determine their level of access. For example, an administrator may have full access to the database, while a regular user may have read-only access.

Encryption. Encryption is the process of encoding data in a way that only authorized parties can read it. Time series databases can use encryption to protect data at rest and in transit.

Authentication and authorization. Authentication ensures that users are who they claim to be, while authorization determines what actions they can perform. Strong authentication and authorization policies can prevent unauthorized access and data breaches.

Network security. Time series databases can be secured by implementing network security measures, such as firewalls, intrusion detection systems and virtual private networks (VPNs). These measures can help protect against external threats and prevent unauthorized access to the database.

Audit trails. These can be used to track who has accessed the database and what changes have been made. This can help detect unauthorized access and prevent data tampering.

The post How to Choose and Model Time Series Databases appeared first on The New Stack.

In the current data-driven business landscape, organizations are heavily dependent on the time series data they collect and the stories it can tell them about what customers want — and are likely to want in the future.

With the right query optimization techniques, organizations can extract valuable insights from their time-series data faster and more efficiently, gaining a competitive edge in their respective markets.

Without those techniques, data queries will be inefficient, data analysis will be time-consuming and inaccurate, and companies will be more likely to fall behind their competitors.

As time-series data grows, the number of queries needed to access it also increases. Optimizing queries involves streamlining the process of accessing and filtering data, as well as reducing the computational resources required to process and analyze large volumes of data. This results in faster data processing, improved data accuracy, and reduced costs associated with data storage and retrieval.

In this article, we’ll share some tips for optimizing time series data queries.

Kinetica is available as a fully managed service in the Cloud. Continued focus on core features and an improved Workbench interface drives growth among enterprises, new tech and government agencies including some of the largest IoT data analytics implementations in the world.

Learn More

The latest from Kinetica

Understanding Time Series Data

Time series data is a collection of data points taken at regular intervals over time. It has a temporal dimension, which means that time is an essential component of the data. This data is collected and used in various industries, such as finance, healthcare, meteorology, and manufacturing.

This data is often high-dimensional, time-stamped, and requires efficient storage and retrieval. Query optimization is crucial to ensure fast and accurate analysis.

The characteristics of time series data can be used to develop models that can capture the patterns in the data and make predictions. Time series models typically use historical data to predict future trends, and they can be used for forecasting, anomaly detection and signal processing.

Query Optimization Techniques

Here are some overall techniques for creating better queries.

Efficient Filtering

Efficient filtering can help reduce the amount of data that needs to be scanned and improve query performance.

One approach to optimizing filtering queries is to use indexing. Indexing by timestamp can improve query performance by allowing the database to quickly locate data within a specific time range. Indexing by tag can similarly improve performance when querying data based on specific metadata associated with the data.

Timestamp indexing is useful when working with data that is primarily queried by time, while tag indexing is helpful for filtering data based on specific attributes, such as sensor location or measurement type.

Query languages are also an essential factor in optimizing time-series data queries. SQL-based query languages like TimescaleDB and PostgreSQL are widely used in relational databases and have been extended to work with time series data.

Non-SQL query languages like InfluxDB have been designed specifically for working with time-series data and offer specialized features like downsampling and rollups.

When optimizing queries, choosing the right filters is also critical, as it will help reduce the amount of data that needs to be scanned. Minimizing the use of regular expressions and optimizing the use of comparison operators can also help improve query performance.

Aggregation

Aggregation involves reducing the amount of data that needs to be processed by combining multiple data points into a single, summary value.

One approach to aggregation is time-based aggregation, which involves grouping data into time intervals such as seconds, minutes, hours, or days. Choosing the right time interval can have a significant impact on query performance and accuracy. For example, choosing too large of an interval may result in a loss of granularity, while choosing too small of an interval may result in unnecessary processing overhead.

Another approach to aggregation is granularity and grouping, which involves grouping data based on common characteristics such as tags or metadata. Choosing the right granularity ensures that the data is accurately represented while also minimizing the amount of data that needs to be processed. Grouping data by tags can also help to identify patterns and trends within the data.

Another technique for optimizing queries, pre-aggregation involves performing aggregation at write time or query time to reduce the amount of data that needs to be processed. This can be useful for data that requires frequent querying or for data that is frequently updated.

Sampling and Compression

Sampling and compression can be used to reduce the amount of data that needs to be stored and queried, thereby improving query performance.

Downsampling is a common technique used to reduce the size of time series data by aggregating it into smaller, more manageable chunks. This can be done by taking a random sample of the data or by aggregating data points over a specific time interval.

The challenge with downsampling is to choose the right sampling rate, which can depend on the granularity of the data and the desired query performance.

Time series compression is another technique used to reduce the storage size of time series data while retaining as much of the original information as possible. Compression algorithms typically work by identifying patterns in the data and replacing them with a smaller representation. Some popular time series compression algorithms include delta encoding, Run-Length Encoding (RLE), Floating Point Compression (FPC) and binary packing.

These techniques are particularly important when dealing with large volumes of data that need to be queried in real-time or near-real-time. Proper optimization not only saves time and resources but also improves the overall performance of the system.

Choose the right compression algorithm based on the specific requirements of the use case. For example, some algorithms may work better for data with a lot of repeated patterns, while others may be more effective for data with a lot of noise or randomness.

Ultimately, the choice of compression algorithm will depend on factors like data volume, query performance requirements and the level of compression needed.

Optimizing Database Schema

A well-designed database schema can improve query performance, reduce storage space, and provide better data quality. Here are some methods for optimizing database schema for time series data

Normalize the data. Normalization involves breaking down the data into smaller, atomic units to reduce redundancy and improve query performance. For time series data, this could involve splitting the data into separate tables based on different time intervals or granularities.

Use appropriate data types. Choosing the right data type for each column can also improve storage efficiency and query performance. For example, using integer data types for numeric values instead of floating-point data types can reduce storage space and improve query performance.

Implement indexing. Indexing is a technique used to speed up data retrieval by creating a separate data structure that stores the values of one or more columns. For time series data, indexing by timestamp or by tag can improve query performance.

Use partitioning. Partitioning involves splitting the data into smaller chunks and storing them on separate physical devices or servers. This can improve query performance by allowing the database to distribute the workload across multiple nodes.

Optimize for compression. Time series data is often highly compressible due to its repetitive nature. Using compression techniques such as delta encoding, run-length encoding or dictionary encoding can significantly reduce storage space and improve query performance.

Tips for Optimizing Queries

By optimizing queries, users can significantly improve their database’s performance and query response times. Here are some essential tips for optimizing queries in time series databases.

Understand query patterns. Users should identify the common query types they perform on their time-series data, such as range queries or aggregation queries. Query profiling and analysis can help identify query patterns and provide insights into optimizing query performance.

Choose the right database engine. There are many time series database engines available, such as TimescaleDB, InfluxDB, and Prometheus, each with unique features and capabilities. Users should evaluate their use case and select the database engine that best fits their needs.

Tune query performance. Monitoring query performance can help identify slow queries and improve their performance by tuning database configuration settings, such as adjusting memory allocation or changing indexing strategies.

Distribute data across nodes through partitioning and sharding. Data can be partitioned by time or tag, and sharding can help distribute data across multiple nodes to improve query performance and reduce latency.

Use caching. Caching can be used to store frequently accessed data and reduce the number of queries needed. This can help to improve query performance and reduce resource usage.

Use appropriate hardware. Use hardware that is optimized for time series data storage and processing. This can include specialized hardware components such as GPUs and FPGAs.

Tips for Writing Efficient Queries

Here are some final tips to help you write queries for time series data that will help retrieve results efficiently and accurately.

Use indexes. Indexes can speed up query execution by allowing the database to quickly locate relevant data. For time series data, indexes can be created on timestamp or tag columns to enable efficient filtering and aggregation.

Minimize data retrieval. Only retrieve the data you need. Use filters to narrow down the results and limit the number of rows returned. This can significantly reduce query execution time and resource usage.

Use appropriate time intervals. Choose the right time intervals for aggregation and downsampling. Using smaller intervals can provide more accurate results, but it can also lead to increased resource usage. Using larger intervals can reduce resource usage but may sacrifice accuracy.

Optimize joins. Avoid joins if possible or minimize the number of joins needed. Joining large tables can significantly slow down query execution.

Use the right data model. Choose a data model that fits the specific needs of your application. For example, if you have a high volume of writes, consider using a column-oriented data model. If you need to perform complex queries, consider using a relational data model.

Avoid using wildcards. Avoid using wildcards like % in your queries, as they can cause the database to perform a full table scan. Instead, use specific criteria in your query to narrow down the results.

Use batch operations. Batch operations will perform multiple operations in a single query, rather than executing multiple individual queries. This can reduce the overhead of query execution and improve performance.

Use prepared statements. Parameterized queries will reduce the overhead of query compilation and execution. Prepared statements allow you to reuse the same query multiple times with different parameter values.

Use the correct data type. Using a data type that’s not optimized for timestamps can lead to slower queries and increased resource usage. For example, use a “timestamp” data type instead of a “datetime” data type.

Use pre-aggregation. Pre-aggregation involves calculating aggregate values ahead of time and storing them in a separate table. This can help to speed up queries that require aggregation by reducing the amount of data that needs to be scanned.

The post How to Optimize Queries for Time Series Data appeared first on The New Stack.

Calyptia recently unveiled Calyptia Core 2.0, which includes a number of key updates to its flagship telemetry pipeline platform. New features include scores of additional processing actions for data in pipelines, an AI assistant that simplifies the user experience for specifying processing needs, and fleet management functionality for controlling processing agents at the edge.

Calyptia Core 2.0 was announced on Observability Day Europe during KubeCon+CloudNativeCon Europe. Observability is the facet of data management concerned with making the transmission of data from sources to backend targets (for analytics or applications) more cost-effective, efficient, and meaningful.

To that end, Calyptia’s new release is integral to its “strategy for our telemetry pipelines approach that optimizes and gives users control of data flowing through a central point, or remotely through a distributed fleet of agents,” commented Eduardo Silva, Calyptia CEO.

The release signifies some of the most important motions occurring in the contemporary data space. The fleet management capabilities coincide with the mounting importance of edge computing and the distributed data ecosystem, while the AI agent typifies the increasing capacity to communicate with data systems via natural language.

Infrastructure enables innovation. HashiCorp provides consistent workflows to provision, secure, connect, and run any infrastructure for any application.

Learn More

The latest from HashiCorp

Fleet Management

The notion of telemetry pipelines is predicated on sending data from sources to targets while processing it between these endpoints to streamline transmissions. Some use cases involve centralizing data prior to sending it to targets via telemetry pipelines that transform it according to business needs. However, there are a number of applications, some of which involve low latency data, in which it’s increasingly viable to send data directly from the edge to targets. Calyptia 2.0 has enhanced features to support situations in which, “I have a couple of agents sitting on edge nodes collecting the data and sending data [directly] to the telemetry pipelines,” Silva said.

Prior to the latest release of the platform, Calyptia users had to write complicated, customized scripts to manage Fluent Bit agents at the edge to perform these duties. Calyptia Core 2.0 contains mechanisms that automate the installation and configuration of the agents required to initiate data transfers for edge deployments. Consequently, it’s now much easier for users to “segment a fleet of agents and integrate different types of telemetry pipelines for different needs of different teams, plus monitoring, plus performance improvement capabilities,” Silva noted.

Processing Functionality

The fleet management facets of Calyptia Core 2.0 expand its chief value proposition of decreasing the amount of data sent from sources to targets while improving its relevance to business objectives. The new transformations it has, which include methods for aggregating, redacting, enriching, and reducing data in flight, compound this value and are convincing on their own, too. Calyptia Core 2.0’s transformations and processing rules allow organizations to maximize the utility of the data transmitted, largely by reducing its size while boosting its usefulness.

According to Silva, “Some customers say that every event they have in a log is huge, but I only need a few fields of that message; I need 20 to 30 percent of that information. So, they go ahead and write a processing rule with our tool, which will treat that message to the state they need that is useful for them.”

The processing capacity of worthy telemetry pipeline solutions effectively whittles down the data that’s transmitted to reduce the amount of data that moves. This decrease in data movement leads to a similar decline in the costs of managing data while increasing the effectiveness of the data that is moved for applications or analytics.  Silva articulated another use case in which, “In the processing phase you can easily discard data. For example, everything that is a DevOps message that was created by a developer, don’t send it to Splunk in production.”

Less Is More

If users require a processing action that’s not already built into Calyptia Core 2.0’s, they can readily write it in a script for a degree of automation in the future. The platform also has an AI agent that’s helpful in this regard and enables “you to write in human text what you need to accomplish and the AI assistant is going to write the processor action for you,” Silva mentioned.

These developments, in addition to the newfound ease of controlling telemetry pipelines at the edge, help Calyptia further fulfill its mission of providing data observability. “Generating more data doesn’t correlate with more value,” Silva cautioned. “It’s the opposite. More data is more noise unless you have a strategy to control that data flow.”

The post Calyptia Core 2.0 Tackles Fleet Management for Observability appeared first on The New Stack.

How do you deploy Kubernetes on the edge in areas of remote connectivity and little to no bandwidth? With low-to-no supervision? How about deploying, updating and securing a fleet of AI-enabled flying robots to pick fruit?

These are the challenges Tevel’s next-gen harvesting solution is up against as it looks to limit food waste and feed more people with less people. As about 30% of the world’s fruit is not picked in time, due to logistical challenges and seasonal labor shortage, Tevel looks to fill that gap with flying autonomous robots already deployed across Italy, California and Washington state.

The smart agriculture or AgTech industry is growing year over year, as a way to increase food production and security alongside the scaling global population.

DevOps leader Itzik Ben Zaken spoke of Tevel’s software plus hardware plus the edge journey during the Kubernetes on the Edge day as part of KubeCon+CloudNativeCon Europe. He was joined by Pedro Oliveira, solutions architect at Spectro Cloud. Together their teams collaborated over a solution.

We break down the sociotechnical challenges and the technical solutions — including Kubernetes on the edge — now.

Spectro Cloud uniquely enables organizations to manage Kubernetes in production, at scale. Our Palette management platform gives effortless control of the full Kubernetes lifecycle, across clouds, data centers, bare metal and edge environments.

Learn More

The latest from Spectro Cloud

The Challenge of Launching a Swarm of Robot Fruit Pickers

First, the setup. Tevel’s autonomous robots are connected via power supply and networking cable to central units that process the sensors and sophisticated AI that searches for, navigates to, and detects the fruits, very gently picking them when ready. The flying robots also act as real-time data agents, gathering information about fruit weight, color grading, ripeness and disease detection. Also, the two-thirds of the year the fruit isn’t locally in season, the multifunctional robots can be deployed to do other things like pruning and precision spraying.

Also looking to help address the global bee shortage, Zaken later told The New Stack that Tevel is working with bio-mimicking pollination tech company BloomX on autonomous pollination of avocado trees.

This is all achieved autonomously. Unsurprisingly, there were many technological challenges to overcome to make the solution happen.

“As much as we want our robots to be autonomous, we want our applications to be autonomous,” Zaken said. “You need something like Kubernetes to do this job because fruits are not waiting for software updates.”

But these Kubernetes clusters are in fields, far away from 5G and antenna towers. These swarms have to work in spite of unstable connectivity, low or no supervision, access difficulties, risk of theft, and being very difficult to deploy.

Tevel’s engineering team is based in Tel Nof, Israel, so they can’t be flying halfway around the world to set anything up or to fix something. That would just be another day of fruit picking wasted. They have to be able to manage the Kubernetes clusters in the field, with no network and connectivity instability. And be able to do it all via remote installation.

“As a startup, you need to run, and run fast, so each developer, at the beginning, had their own robot,” Zaken said. At first, they each needed to connect an SSH or Secure Shell network communication protocol and manually scrape to remote control each robot, with an x-server to get video, using bash scripts to control the robots remotely.

What they ended up with was different setups for each engineer and each host. The robot setup was manual with the operating system and applications, and it was painful, Zaken said, giving the whole team a bad case of “works on my machine” syndrome.

This created a high onboarding cost for both the devs and ops teams, and “it was impossible to operate multiple drones. It was too many screens,” he continued.

Initial Challenges of Kubernetes on the Edge

About four years ago, the Tevel team decided to embrace containers. They implemented Docker for easier setup and application, and to “freeze our setup.” The Tevel team built a small web user interface to control everything. But they still had a lot of challenges.

“When you’re handling containers manually, it’s difficult. Our images were huge, heavy, and the build was difficult,” Zaken said. At the time, deployment was separate between the deployment station and each robot. Even with containers, the robots were still hard to operate.

So they decided to try Kubernetes. Tevel divided services into different containers. Finally, the deployment lifecycle streamlined, the robots (and their humans) were able to embrace DevOps by combining deployments with the operation station. Immediately, with microservices architecture, the images became slim and thin. They were also able to add service monitoring and controls to the web UI.

“Multi-robot operation was now a breeze,” Zaken said.

But they initially found a lot more challenges with Kubernetes on the edge, at their scale of millions of metric tons. It was complex to manage multiple host installations at scale, he said, and gathering statuses for all assets was too manual. They were still unable to be sure the cluster was running all the time, when far away from antennas, he continued, and operating multiple platforms was just plain hard.

Since security work is never finished, extra security needs were required like operating system compliance and manual encryption — which Zaken noted is particularly painful at the edge.

Most importantly, they wanted to reduce the cognitive load on their team, and to streamline a more out-of-the-box developer experience.

“At Tevel, we wanted to focus on the application, on picking fruits, not the update system,” he explained.

Tevel Partners with Spectro Cloud on the Edge

Tevel began working with Spectro Cloud to find these next solutions. The partnership zeroed in on four needs shared by the hundreds of these geographically distributed, multi-purpose robots:

• Easy provisioning of operations
• Functional when disconnected
• Zero risk remote upgrades
• Full-stack security

This stack includes the ground control station — a Kubernetes cluster that is run by an operator and can control any number of bins. That station is wirelessly connected to bins that can then manage up to eight multipurpose robots for fruit picking or pruning.

In order to manage Kubernetes clusters at scale at the edge, they adopted Spectro Cloud Palette Kubernetes lifecycle management platform, which works across data centers, public clouds, the edge and a combination of these options.

On the edge, Oliveira from Spectro Cloud explained at KubeCon, Palette manages the highly distributed architecture, with each cluster running its own Palette agent, separating the management plane from the control plane. This was designed specifically for such disconnected use cases with no bandwidth.

“Like Itzik was saying, they don’t want to be playing around with OS dependencies, having to install Nvidia dependencies, kernel dependencies, all the way to the application. And that includes also all the integration in between,” Oliveira explained. “With Palette, you specify the desired state of your cluster, which we call a cluster profile.”

For about 90% of edge use cases, Oliveira said, connectivity and security are always an issue, which makes provisioning — setting up and deploying a device — a persistent challenge.

Within Palette, you specify a cluster profile. Then you apply that cluster profile to a cluster. And then the cluster declaratively reconciles and manages itself with the desired configuration and policies. Spectro Cloud calls this easy provisioning, at the edge.

Tevel may have very short notice that fruit is ready for the picking, so they needed the ability to get robots shipped to and provisioned at distribution centers that aren’t set up with the right equipment or staffed with the right technical skills.

To this end, Spectro Cloud’s Palette uses Kairos, the open source, Linux meta-distribution for edge, to build an immutable operating system and Kubernetes images — in Tevel’s case, based on Ubuntu and K3s. K3s is a Kubernetes distribution designed for the edge and the Internet of Things — unattended, resource-constrained production workloads in remote locations that require high availability.

“Once you boot up and install your device, you’ve only got to ship it out and the operator has to power it on and it connects. From then on, he can register the device with Palette,” Oliveira explained. Once that device is registered, they can build a cluster. All of Tevel’s operations are run through Spectro Cloud Palette.

Out in the fields, the robotic devices come with QR codes that trigger bin and edge device registration via Palette, including:

• Edge machine ID
• Device name
• Farm ID
• Deployment region for multi-agency

But not all the operators in these rural environments have the connectivity or even the screens to register these devices. Or to update registration if the hardware fails, requiring a quick swap out.

For these cases of connection disruption, there is an added requirement of headless, zero-touch provisioning, without a registration, with all the above information provided in advance, alongside an edge token that is authenticated against Palette. Then the device is registered automatically with Palette, even if not online.

Another challenge is when the environment is fully disconnected — on what is called a gray site, or an air-gapped deployment. When there’s no connectivity between the field and Tevel’s headquarters, how can Tevel not only maintain these robots, but even know if the functionality is still there? Because Palette is intentionally built with distributed architecture, the edge nodes are separated from the control plane and management plane.

“Once the cluster is running and everything is going, the cluster is managing itself. So even if there’s no connectivity back to the management platform, the node, the device, and the cluster, will still maintain its status, its applications, its configurations, and all of its policies as well,” Oliveira explained.

Kairos is a container-based image that splits a disk into A/B or active and inactive partitions. If an upgrade fails, it automatically reverts back to the previous configuration, so the upgrade doesn’t break the workflow. This is referred to as a zero-risk upgrade.

With Kairos, the operating system is also immutable, or unable to be changed, with read-only roots to provide extra security, so nothing can be installed on an individual cluster or device.

With these robots, the cost of burglary is a consideration beyond the device but to the intellectual property too, particularly when they are offline. It wouldn’t be scalable to have to generate YubiKeys for each of 60 bins (the groups of up to eight robots.)

The Tevel-Spectro Cloud collaboration decided to use a Trusted Platform Module or TPM to deliver persistent data encryption. Oliveira explained that the TPM generates unique one-way hash keys that are saved into the hardware. Both a TPM and a key management service (KMS) are required to boot. Every time the device boots, it challenges the TPM, and will only then turn on if that challenge is successful. This means that the disks remain encrypted if a device is stolen.

Zaken says Tevel’s mission is to help feed the world. They aim to achieve this by helping farmers be more profitable by always being able to pick every piece of fruit on time, any time of day, while also gaining real-time data on each piece picked. The Tevel team doesn’t want to get distracted by hardware and connectivity issues. They want to focus on increasing the overall efficiency and reliability of their process.

This tech setup “reduced the time that we spent on building and managing clusters by ourselves,” Zaken said. “And it frees our engineers’ time to be focused on innovation” around robot fruit picking.

Introducing SENA: Secure Edge-Native Architecture

Next step is they want to reduce their infrastructure cost, getting rid of the ground control station level.

And of course are looking to increase security throughout the stack, implementing the Secure Edge-Native Architecture (SENA) framework announced at KubeCon earlier this week. SENA is the result of a collaboration between Spectro Cloud and Intel which applies zero trust principles for deep-root security at all layers of edge computing environments. SENA integrates the best-of-edge technologies from Intel, Spectro Cloud and open source, including Intel Smart Edge, Spectro Cloud’s Palette, and Kairos.

SENA offers a holistic pathway for advanced security across the full-edge lifecycle from silicon to app:

• Deploying trusted hardware – how to know a device hasn’t been tampered with — device onboarding and authentication
• Provisioning a verified software stack to the hardware – how to know that the operating system, Kubernetes, and application are all as intended (software bill of materials or SBOM) and free of vulnerabilities (across the supply chain levels or SLSA)
• Operating runtime securely – including verifying the boot, image immutability, confidential computing, encryption in use and in transit, and workload isolation
• Managing the edge lifecycle – including monitoring, patching, upgrades, zero trust role-based access control (RBAC), and self-healing reconciliation loops

By applying these next steps for security on the edge, Tevel can focus on making sure each ripe fruit is picked and every possible side job for the robots is optimized.

Check back often for all things KubeCon+CloudNativeCon Europe 2023. The New Stack are your eyes and ears on the ground in Amsterdam!

The post Fruit-Picking Robots Powered by Kubernetes on the Edge appeared first on The New Stack.

Kubernetes has a pretty fearsome reputation for complexity in its own right (as we’ve discussed before). Learning it for the first time and setting up your first cluster, deploying your first application stack… it can be painful.

But as any seasoned operator will tell you, it’s when you expand into running Kubernetes in production at scale that you come across the real pain!

Let’s delve into three of the most common “growing pains” that we’ve seen in the field:

• Developer productivity
• Multicluster headaches
• The edge learning curve

We’ll not only explore the pain, but show you ways to sidestep these pitfalls.

Spectro Cloud uniquely enables organizations to manage Kubernetes in production, at scale. Our Palette management platform gives effortless control of the full Kubernetes lifecycle, across clouds, data centers, bare metal and edge environments.

Learn More

The latest from Spectro Cloud

Pain 1: Developer Productivity

Infrastructure doesn’t exist for its own sake. As an operations team, the clusters you create are there to provide application delivery for the dev teams you support.

Despite the popularity of the term “DevOps,” most developers don’t have the skill set to be experts in cloud native infrastructure or Kubernetes. They would much rather be coding features than managing infrastructure (as we have explored in this blog post).

Developers just want to consume infrastructure elements such as Kubernetes clusters, and they have little tolerance for delays and hurdles in their way. Unfortunately, it’s not always easy to give them what they want.

Let’s take a simple example: running a test suite on a new release of application code.

The developer wants a pristine, clean cluster — possibly multiple clusters, running specific versions of Kubernetes and other software. This is vital so they get accurate testing that predicts production behavior. We all know that it’s impossible to mirror a production infrastructure setup on a local laptop, as tempting as it is.

If a test fails for a simple reason, the dev might want to repeat the CI/CD pipeline almost immediately.

But you know it’s not that easy.

Firing up a new cluster takes work, costs money, and even if you have the capacity to jump right on the request, it also takes time. Which means your developers are kept waiting.

This is a real conundrum. Now imagine it happening across dozens of dev teams pushing multiple code pipelines per day.

Giving Devs What They Need with Virtual Clusters

So, how do you deal with it? One answer is virtual clusters. In a virtual cluster setup, the infrastructure team keeps host clusters stable and under their full control while giving developers virtual clusters. Virtual clusters are isolated, presenting zero risk to the underlying infrastructure, and take next to no time or resources to fire up, meaning you can give every team their own clusters to play with.

This solution is based on the open source vcluster project, but at Spectro Cloud we’ve made this technology enterprise-ready and easy to consume. That is, we’ve added a sleek UI, implemented a fully declarative approach with Cluster API behind the scenes, connected all this to role-based access controls and presented it as an easy-to-consume SaaS solution.

Using Palette, developers can order their own clusters created specifically for them based on cluster profiles, defined by the infrastructure group. Each team can have their own playground. Of course the whole process can be fully automated using Terraform provider or REST API calls. With this approach, granular control remains in the hands of the infrastructure team, but developers now have their freedom and ability to test their code in prod-like conditions.

Pain 2: Multicluster Headaches

Everyone starts with one Kubernetes cluster. But few teams today stay that way.

This number quickly grows to three when you split development, staging and production environment clusters.

And from there? Well, our research found that already half of those using Kubernetes in production have more than 10 clusters. Eighty percent expect to increase the number or size of their clusters in the next year.

You can probably manage a couple of clusters manually using kubectl, k9s and other open source tooling. But once you grow to a couple dozen clusters, managing Kubernetes becomes overwhelming.

How can we solve this problem?

The foundational principle of multicluster management: You shouldn’t be manually using kubectl and connecting to your clusters individually. Instead, you describe their configuration in a declarative approach.

That “future state” description should cover the entire cluster, from its infrastructure to the application workloads that run on top. In other words, you should be able to recreate your whole cluster stack from scratch using only its description, without manual intervention.

Multicluster, Multicloud, Multienvironment

As you look ahead to growing numbers of clusters, it pays to be cloud-agnostic from the start.

It’s tempting to stay with just one public cloud provider. It’s the easiest path: only one product set and terminology to learn. And cloud providers normally do their best to lock you in, often financially incentivizing loyalty.

But there are many reasons why you might find yourself in a multicloud environment. Mergers and acquisitions sometimes unexpectedly bring another provider in house. You might want to use multiple clouds to access specific features, or to spread risk and improve availability.

A number of companies we’ve seen are also using a hybrid architectural approach to deploy Kubernetes: some legacy data-center-based deployment applications coupled with cloud-based ones.

Whenever you’re dealing with different siloes of Kubernetes, what really helps is a single pane of glass. If you’re using multiple cloud providers or are on premises plus using a cloud provider, you need tooling that enables you to make similar deployments to various environments. It helps achieve standardization and simplifies your overall organizational infrastructure.

Pain 3: The Edge Learning Curve

From the data center and cloud, you might start looking even further afield: to the edge.

Organizations are increasingly adopting edge computing to put applications right where they add value: in restaurants, factories and other remote locations.

But edge presents unique challenges. The hardware is often low power: Your clusters might be single-node devices. The connectivity to the site may be low bandwidth or intermittent, making remote management difficult. There’s a whole new frontier of security to consider, protecting against hardware tampering.

And the killer: When we’re talking about restaurant chains or industrial buildings, compute might need to be deployed to hundreds or thousands of sites. There won’t be a Kubernetes expert at each site — or even a regular IT guy — to help onboard new devices or fix any configuration issues locally.

These are big challenges, but there are solutions to help you.

Low-touch or no-touch onboarding technologies mean that someone without IT or Kubernetes knowledge should be able to take a device, plug in power and ethernet, and watch provisioning complete automatically, without any human intervention whatsoever. (Check out a demo of that right here.)

One of the prerequisites for this deployment is an ability to provision your whole cluster from scratch in one go, including operating system provisioning on the underlying device. When you can do such a provisioning in a declarative way, this significantly simplifies mass-scale deployment. The idea here is that provisioning should start automatically and build the system up to a point when it’s capable of reaching HQ, from where you’d be managing the deployment centrally.

Once your cluster is ready, you can further mitigate risk if your OS is immutable. It has fewer elements to break, so it provides a more reliable foundation for your clusters. So, pay attention to which OS you’re choosing. There’s a number of both open source and commercial offerings available on the market. Check out this comparison to get started.

Facing Challenges? You’re Not Alone.

As a platform team, you will almost certainly face these challenges as you scale Kubernetes in production. They’re daunting, for sure. But there are approaches you can take to be successful.

The cloud native, open source Kubernetes ecosystem is huge, and you know you’re not alone. There are many different projects out there that you can apply. And there’s a wealth of community experience you can draw on too at events like KubeCon.

But perhaps the biggest tip we can offer is this: If you’re building for scale, don’t DIY.

By all means start with DIY to learn and prove the concept. But you should know that the “do it yourself” path only becomes harder with time. We have moved on from needing to “do Kubernetes the hard way.”

So what’s next? If you’re interested in diving deeper into the challenges you can expect to see as your Kubernetes use grows, join me in our webinar on April 27. Register here!

The post Three Common Kubernetes Challenges and How to Solve Them appeared first on The New Stack.

The notion of internal developer platforms (IDPs) is very popular at the present time. Software engineering teams of all types have it at the top of their wishlist. Teams within large organizations are able to devote resources exclusively for the development and maintenance of IDPs, but the demand is ubiquitous.

Based on observations at the KubeCon + CloudNativeCon NA 2022 floor, several leading surveys, and general feedback — we concluded that the Kubernetes community can greatly benefit from platforms that simplify its adoption and day-to-day usage.

KubeCon + CloudNativeCon conferences gather adopters and technologists to further the education and advancement of cloud native computing. The vendor-neutral events feature domain experts and key maintainers behind popular projects like Kubernetes, Prometheus, Envoy, CoreDNS, containerd and more.

Learn More

The latest from KubeCon + CloudNativeCon

Kubernetes Is Powerful but Complex

Since the nature of the problem solved by Kubernetes is a complex one, Kubernetes itself can be difficult to manage. The technology handles many important tasks involved in running an application such as persistent volumes, load balancing, secrets and authentication, service discovery, auto-scaling and others. All have the intent of liberating application developers. Instead, we find that application developers end up being tasked with knowing Kubernetes primitives, which can be a significant cognitive overhead. Over time, so many new features have been built into Kubernetes that it has started to work in myriad ways, which is demonstrated in this excellent talk by Viktor Farcic.

In the past, we have written about how DevOps practices shape over time. Our observation is that ― many small factors add up to bigger outcomes. The use of platforms for software engineering teams is no different. Small, yet significant outcomes for engineers bubble up as important results for product teams. In turn, this leads to meaningful development for the business units, culminating in success for the company as a whole. For those in the business of delivering software products to their customers, this paradigm can be illustrated using the pyramid below.

What Should Teams Shoot for?

Individual software teams need to be equipped with the right kind of tools and processes in order to be able to deliver the desired results. Both velocity and accuracy are paramount in order to release functionally. Defining the right kind of Sservice Level Objectives (SLOs) is critical in meeting availability and reliability requirements. SLOs can take many forms. It can be defined in terms of a percentage for uptime, in units of seconds or milliseconds when defining (un)acceptable latency or by volume when defined for throughput. In the event that SLOs are not met, engineers should have sufficient knowledge and transparency to be able to investigate and ascertain the root cause. Teams should be able to collaborate with each other and work together to inspect problematic areas, especially those that are cross-functional.

Upon such a foundation of sound engineering practices, value addition can happen downstream. High-quality software can be engineered and delivered by teams designed to be dynamic and agile. Teams should be able to remove opacity in all areas of malformed operations, including application errors, security misconfiguration and vulnerabilities — in addition to latency and reliability issues on production.

Armed with these capabilities, engineering teams can promise faster development cycles. By consuming environments and frameworks that are standardized.

Platforms Are Everywhere

Platforms are not limited to cloud computing. They can be used to improve the workflow of developers involved in all kinds of stacks. For example,  Blockchain developers have options such as Avalanche. Avalanche provides primitives that allow developers to then build decentralized apps utilizing exchanges, platforms or contracts as needed. Similarly, Fermyon is positioned as a platform for WASM (WebAssembly). With just a few commands, a developer can spin up a WASM application and deploy it. Their combination of CLI and web UI allow developers and operators to work together on the platform to manage applications. Another example of a platform for WASM is Cosmonic.

Overall, a platform is intended to help organizations to build better software faster, with improved collaboration, quality and scalability.

A Platform for Kubernetes

Kubernetes is an emerging technology that can greatly benefit from having a paved path to production associated with it. Using a platform, application developers can simplify the application deployment process by being able to deploy to any remote instance without having to engage in manual configuration for various clusters.

A platform consumed as a PaaS tool is also of great value when services have to be integrated with applications, especially when working with Kubernetes. Built-in support for databases, messaging and monitoring saves enormous amounts of time and liberate developers from significant toil. It also improves the production parity of applications by making the same grade of services available on all remote instances. This increased level of automation is useful in accurate deployments and can help scale applications on demand.

Conclusion

We believe that the Kubernetes community can greatly benefit from having platforms built to abstract away some inherent complexity. A definitive set of tools, processes, and services that surround Kubernetes clusters in order to establish developer self-service is key in helping improve the adoption of Kubernetes. With this, software engineering teams will be able to build, test and deploy their applications with efficiency and effectiveness.

The Cloud Foundry Foundation will be sponsoring KubeCon + CloudNativeCon Europe 2023 this April in Amsterdam. We’ll be hosting a booth with our contributors present to discuss platforming on Kubernetes. We invite you to please stop by for a chat and check out Korifi — The Cloud Native Platform For Cloud Foundry.

The post A Platform for Kubernetes appeared first on The New Stack.

Time for a home truth about doing edge computing: The one thing most likely to kill your edge Kubernetes project is a brown cardboard box.

“Huh?” I hear you say. Well, grab your popcorn and lean in because things are about to get real.

Does the Cloud Native Community Know Enough about Logistics?

We’ve spent more than a decade containerizing and moving to the cloud (and we’re still at it).

The siren call driving this mission is speed of execution: Focus on your business case and your software, and don’t worry about infrastructure. Software is eating the world; it’s where the value is. It’s where the innovation is. It’s where you should pay the most attention.

By infrastructure, I mean all those servers, networks and storage that’s abstracted away, taken care of by hardworking people doing magic that you no longer have to worry about.

Spectro Cloud uniquely enables organizations to manage Kubernetes in production, at scale. Our Palette management platform gives effortless control of the full Kubernetes lifecycle, across clouds, data centers, bare metal and edge environments.

Learn More

The latest from Spectro Cloud

We’ve all been so focused on beautiful GitOps, software supply chains, Infrastructure as Code, and a raft of other things. When it comes to our apps in the cloud, infrastructure is just something at the end of an API. We snap our fingers, and it’s there.

For edge, suddenly we have to care about nonabstract things again. The real thing: hardware.

There are groups out there that never forgot how important hardware is, of course.

• They’re the communication service providers building base stations on remote Scottish islands and laying undersea cable.
• They’re the IoT companies worrying about ingress protection ratings when they’re bolting toughened sensor boxes to oil rigs.
• They’re the SD-WAN (software-defined wide-area network) or SASE (secure access service edge) companies stringing cable between racks in local points of presence.

Edge Kubernetes is, in a sense, a marriage of our treasured cloud native paradigms and this good ol’ fashioned iron.

So what can we learn from those groups that know all about the hardware?

Hardware Is Hard and Failure Is the Norm, Not the Exception

Anyone involved with maintaining servers or fleets of devices at the edge will tell you the same thing: Hardware failures are business as usual.

It boils down to statistics. Take a standard Intel NUC (Next Unit of Computing) or compact server typically targeted for use at the edge. It’s composed of several discrete components: CPU, RAM, hard drive or SSD, fans, power supply and more, each with minimal redundancy, unlike what we might be used to in a data center.

Now imagine a fleet of 10,000 of these small devices. That means 10,000 or more of each of those components, each of which has a rated MTBF (mean time between failure).

Making the Best of MTBF

MTBF becomes one of the key variables in the equation of whether your edge K8s deployment makes money or loses it, in downtime and the cost of hardware and labor. So it pays for you to understand how to make the most of it.

Sure, you can procure components with better reliability, but that increases cost and may not be feasible for your design requirements. You can build in redundancy, but that again increases cost and may move you beyond the space, power and cooling envelope for your edge location.

You can do what you can to control the environment. MTBF might be worsened by suboptimal operating conditions in hostile edge locations. Maybe the retail store manager likes to pile paperwork on top of the mini server and blocks the cooling vents. Tell them not to do that, of course, but humans will be humans.

So you should also plan to monitor mundane but crucial hardware components like fans and power supply units right along with the applications running on your edge stack. They’re just as crucial. Boxes running hot should generate alerts and be used to take proactive measures. These things should be included in any holistic edge Kubernetes system and come out of the box along with sensible alerting rules.

If you expect your edge device deployment to be ‘one and done,’ you’re in for a rude awakening.

Software can affect MTBF, too. Mundane things such as how the Kubernetes stack reads and writes for behaviors like logging can quickly kill an SSD, and tuning app behavior can’t completely avoid it.

The upshot is this: Although the odds of a single unit failing on a given day are very low, when there are enough units, you can practically guarantee that one of them will fall over … today, tomorrow and the next day. It would be normal to have several device failures every single week.

So if you expect your edge device deployment to be “one and done,” you’re in for a rude awakening.

Planning for hardware failure, and how to handle that failure efficiently, is the name of the game if your return-on-investment calculation is to look good.

Logistics and Spares-Holding: More Complex than You’d Think

To be great at handling failure requires thinking logistics, and specifically to answer the question: When a node fails, how do I get a spare swapped in and running, fast?

It’s obviously not cost effective to keep a full set of spare hardware for every device you’ve deployed, and certainly not on site for a hot swap. Your finance people would be furious at tying up that much money in inventory, and often there isn’t space or security at remote sites.

So we instead need to store a smaller pool of spares in inventory somewhere else — central warehouses or regional distribution centers — and ship them to the affected edge site when needed.

I’m not a Markov model expert, but the optimal amount of hardware required to be stored centrally/regionally can be calculated primarily based on the anticipated failure rates of the devices and the MTTR (mean time to restore/recovery). A more efficient process for replacing devices lowers the MTTR, which in turn lowers the number of spares required in warehousing.

No matter how many spares you keep, they can’t be boxed fresh from the supplier either — although you may be used to doing so for end-user devices or data-center compute and networking equipment, which can be set up by an on-site team at the destination.

Edge sites like restaurants or wind turbines won’t have an IT expert there to install operating systems and configure the K8s stack on site, and you certainly don’t want your high-salaried K8s ninjas driving for six hours to a different remote facility every day with a keychain full of USB sticks and a keyboard and monitor on the back seat.

The hardware needs to be prepared to some extent somewhere centrally, preloaded with an operating system and software so that when it’s powered on, it can join the K8s cluster and start working.

Still with Me? There’s More

But even here we’re only just scratching the surface of the complexity you need to engineer for when building a spares process for a production edge Kubernetes deployment.

Different deployment destinations (for example, a small retail branch vs. a megastore) might have different workloads, meaning they are issued with a different device variant and a different software stack.

Certain elements of the device configuration will always be site-specific, such as secrets, IP addresses and network configurations.

Indeed, regardless of such hardware variation, certain elements of the device configuration will always be site-specific: secrets, IP addresses and network configurations, location-related user data on the device, etc.

As a result, you simply can’t fully stage the spare device until you know exactly which site that device is going to — and by definition that will be after a mission-critical edge device has already failed and the pressure is on.

So we’re building up to a list of requirements:

• If failures happen daily, the staging process and tooling for device preparation needs to be simple and lightweight and able to be automated.
• If spare provisioning is only triggered by a mission-critical failure, the whole process has to happen quickly and consistently, with few manual steps.
• If we can’t send engineers to each site, we need the ability to seed devices securely with the information needed for them to join a cluster — and to remotely troubleshoot and observe the status of devices after they are powered up at a site, before and during attempting to join or form a cluster.
• If we are keeping a ratio of 1:x spares per live site, we need the ability to provide site-specific information as late in the logistics chain as possible, to enable all devices that are stored to be homogeneous in terms of software configuration.
• If we can’t rely on totally interchangeable hardware (because we have variants), we shouldn’t require pre-provisioning of different roles, such as a master node or a worker node, labels for GPUs, etc. This should be detected automatically and assigned dynamically when a device arrives at a site and is ready to form or join a cluster. Without this, it requires additional human involvement either during staging or during onboarding, and it introduces more room for error.

Securing Your Brown Cardboard Boxes

At Spectro Cloud, we talk a lot about Kubernetes security, and in particular edge device security. We imagine deliberate tampering and attack of edge devices in situ, by masked assailants.

But as anyone who has ordered an expensive present for a loved one will know, the biggest risk is from supply chain “shrinkage”: when a brown box mysteriously drops off the radar somewhere after it’s dispatched.

You need to account for this, and not only by choosing a trustworthy logistics partner to operate your warehouse and deliver your parcels.

Your edge devices — especially since they’ve been preconfigured with data! — need to be encrypted, with the use of Trusted Platform Modules to secure boot, remote wipe capabilities and all the other protection capabilities you can lay your hands on.

A lost or stolen device should be red-flagged so it can’t join the corporate cluster, and an assailant certainly shouldn’t just be able to carry your data away under their arm.

Wake-Up Call Officially Over

If only meatspace was as easy to orchestrate as our cloud native abstractions, eh? But the reality is that edge is very physical. It’s about humming boxes installed in cupboards, under desks, in thousands of forgotten locations that only a whistling delivery driver or a guy in a hi-viz vest will ever see.

If your edge project is ever to move successfully from a pilot to mission-critical production at scale, you need to think about failing fans, piles of boxes in warehouses and everything that needs to happen to get them from A to B when every second counts.

Need Some Help?

We’ve been building features into our Palette Edge Kubernetes management platform to tackle these kinds of scenarios.

For example, Palette offers persistent data encryption and immutable operating systems to address security concerns such as tampering and theft.

Most excitingly, we’ve made big advances in staging and provisioning of edge devices at remote edge sites through features such as auto-registration for headless deployment, low-touch QR code onboarding, and network operations center (NOC)-style remote monitoring. For a quick overview of how all that works, watch my colleague Justin Barksdale walk you through it in this 25-minute video. It’s pretty cool.

If you’d like to talk about what we’ve learned, and your own edge projects, give us a shout. Or to read more about edge Kubernetes, check out our edge resource hub.

The post Why a Cardboard Box Is the Biggest Enemy of Edge Kubernetes appeared first on The New Stack.

Frontend developer platform Vercel will support Shopify’s Remix on its cloud platform, after investing in the framework via research and development. It’s part of Vercel’s effort to broadly support JavaScript frameworks, said Lee Robinson, Vercel’s vice president of developer experience.

“We support 35 frameworks,” Robinson told The New Stack. “Our goal is to try to support any web framework that you want deployed on the internet. So when we were looking at our metrics, we noticed a lot of people were starting to deploy — of course, Next.js — and Remix, versus maybe the traditional Create React App-style application, which is another way of deploying your React apps.”

Vercel said the investment will allow developers to deploy and scale Remix apps of any size or complexity.

Remix: A Short History

Vercel created and maintains Next.js, another open source React framework which is often positioned as a competitor to Remix. React is a JavaScript library for creating user interfaces (UIs) for single-page applications for web and mobile. React frameworks build upon this library. Remix is a React framework. It’s edge-first and full-stack, meaning it can be used to develop both frontend and backend. Created by Michael Jackson, an ex-Twitter engineer, and Ryan Florence, a software engineer who led React Training, it was released in Oct. 2020. They open sourced it under the MIT license in October 2021 and it was acquired a year later by Shopify.

React-based frameworks are built on top of the library, although Remix is less tightly coupled to it than Next.js, according to Facundo Giuliani, a developer relations engineer at Storyblok.

“Both frameworks were created on top of React, but Remix tries to decouple itself from it,” Giuliani wrote last year. “We can see that Remix provides higher levels of abstraction. Also, different Remix community members have been working on different implementations using other frameworks, like Vue.js, Angular and Svelte. Next.js depends on React, and there is no plan to change this at the moment.”

Vercel’s Investment in Remix

Vercel has been working with the Remix team to improve the framework, according to Robinson.

“Our team did contribute actual upstream changes to the repository and worked with the core team there on some suggestions for where to take it in the future,” Robinson said.

One of the suggestions came from customers who used Remix. They wanted to use Vercel edge functions with the framework. Remix was designed to focus on single runtime, he said.

“It didn’t have the ability to be more granular about wanting to use different types of runtimes for different pages. So with Vercel, for example, we have serverless and edge functions — our two compute products that we offer. You can use Node.js if you want or you can use our edge product,” he said.

Vercel contributed changes and worked with the Remix team so that the core of Remix would support multiple runtimes with serverless and edge functions. Similarly, Vercel wanted Remix to be able to break down the serverless apps into multiple separate pieces, rather than one bundle, so that when it’s deployed to a provider like Vercel, each piece could use a serverless or edge function.

“If I have 1,000 routes in my application, you’re going to run into some limits if you try to put that all into one bundle,” Robinson said. “With Remix on Vercel, it can scale to thousands and thousands of routes, because it gets broken apart into these multiple bundles. For hobby developers or small or medium-sized businesses, they love that that compute is able to scale up or down and they pay for only what they use.”

Other functions supported by the integration with Vercel include:

• Streaming SSR, which allows developers to incrementally render parts of the UI to the client;
• Unlocking serverless actions;
• Stale-while-revalidating caching support, which means a request to read a value is served immediately from a cache; and
• Support for larger apps and bundles.

Vercel posted a demo of Remix running on its platform to show the integrations at work.

Robinson said one thing Remix does really well is to help developers think about and take advantage of the web platform.

“What that means in the day-to-day is, rather than having framework-specific APIs, it’s having a framework that builds on top of the web platform itself, and it uses web APIs like request, response and fetch that are portable and interchangeable between different frameworks,” Robinson said. “For developers, this is a really powerful and exciting thing because if I learned how to use the web, fetch API, I can use Next.js or Remix or Sveltekit. And once I learned this knowledge, I can kind of write it in multiple different places… The more that developers can learn pieces and have those be interoperable and shareable between different frameworks, I think is a really great outcome for the web.”

The post Vercel Adds Remix: Integration Supports Larger Apps appeared first on The New Stack.

There’s always a lot going on in the world of Kubernetes. But the eve of a big community gathering like KubeCon gives us the opportunity to take stock of how far we’ve come — and where we’re going next.

I’ve identified five trends that our industry has to tackle right now, which will see play out at KubeCon EU Amsterdam.

1: Edge Means Business

All the indicators are that edge Kubernetes has finally hit the mainstream consciousness. The big analyst houses are publishing their first dedicated reports and vendor rankings. There are more articles, events, and bright new ideas from vendors.

This is all great, of course: the more discussion and collaboration, the stronger the community, even if it means having to wade through more buzzwords.

Importantly, there’s also meaningful progress toward solving for the real-world issues of commercial adoption in production. Issues like:

• How can you truly secure a device and its workloads when it’s unattended in the field?
• How can you keep supply chain costs and errors under control when you’re having to stage and ship new and replacement edge devices regularly for hundreds of locations?
• How can you ensure availability and realistically perform Day 2 tasks like patching when devices are remote and have intermittent connectivity?
• How can you optimize K8s performance for power-constrained edge hardware?

Answering these questions takes a village, which is why you’ll see more and more joint initiatives like our collaboration with Canonical (and spoiler alert: we may have another announcement at KubeCon, so look out for it).

But the real progress is made in the partnership between vendors and the talented engineers working customer-side. That’s what you’ll hear when you attend our session at Edge Day with Israeli agritech startup Tevel Aerobotics.

They’re putting Kubernetes on fruit-picking drones to run real-time AI computer vision workloads. It’s super cool stuff. It’s also a trial by fire for any tech stack, between the razor-thin economics of the agriculture industry, the security and operational challenges of putting hardware literally in the field, and the need for outstanding application availability and performance. So if you’re working on edge yourself, don’t miss this talk.

KubeCon + CloudNativeCon conferences gather adopters and technologists to further the education and advancement of cloud native computing. The vendor-neutral events feature domain experts and key maintainers behind popular projects like Kubernetes, Prometheus, Envoy, CoreDNS, containerd and more.

Learn More

The latest from KubeCon + CloudNativeCon

2: More than Ever, the Developer Experience Matters

While ironically containers emerged as a technology to help developers create more portable applications, Kubernetes is still a set of infrastructure constructs used by operations teams. For years the focus in the K8s ecosystem has been helping ops folks do their jobs more effectively.

But in the world of constantly “shifting left”, that’s no longer enough — developers need to be part of the world of Kubernetes too. Only without having to deal with Kubernetes itself!

Starting with the hype around platform engineering last year, and a bit of a community backlash against the complexity inherent in Kubernetes, we’re noticing a sudden awareness that there’s a second partner in this marriage: the application developer, whose job it is to build applications for, and push applications to, K8s clusters.

Now the community is realizing that frustrated, unproductive developers are a drain on their businesses and we collectively need to help them abstract away some of the complexity of K8s infrastructure that they shouldn’t really have to deal with. Whether you call it ZeroOps or something else, it’s all about the developer experience.

Although several vendors are making a play for this space (including us, we have to admit, with our Palette Dev Engine), it feels like early days — and this is where events like KubeCon are invaluable for helping clear the air and share different perspectives.

3: Bare Metal Marches on

Through our work at Spectro Cloud with customers like Super League Gaming and partners like Cox Edge and Canonical, it’s clear to us that bare metal Kubernetes is not just here to stay, but making great strides in adoption beyond the kind of niche use cases it used to be known for, like AI/ML workloads.

One motivation might be economics — as budget screws are tightened, CIOs are more reluctant to pay the “V tax” where there might be an alternative (especially with fears that Broadcom will ratchet up VMware’s pricing). And if bare metal has a performance advantage, that’s all the better.

Edge is another driver. When devices get smaller, the overhead of a hypervisor becomes infeasible (both in terms of resource requirements and license costs).

And while many started out running Kubernetes on top of virtualized infrastructure, It’s now even possible to flip the script entirely and bring your existing VM-based workloads to run on Kubernetes, through projects like KubeVirt.

Put all those trends together and we’re poised for a wholesale shift away from the world of virtualization. We’re already seeing more announcements about bare metal capabilities, and we’ll see more success stories in the coming months, too.

If you’re working on moving to bare metal, here are some tips.

4: Every Dollar (or Euro) Counts

Compared to the glory days of early 2022, the business landscape is suddenly looking rather bleak. Hundreds of thousands of tech workers have been laid off. Inflation is high. Investors are shutting off the free-money tap. Belts are being tightened. Banks are failing. Fear is in the air.

These events have a ripple effect, and they will hit every organization that uses Kubernetes.

There’s likely to be more scrutiny over costs. A recent report from Sysdig identified the huge waste in Kubernetes resources. Do you have visibility over all your clusters, and the ability to report on cost and manage capacity effectively? Are you able to optimize your workloads across multiple clouds to streamline budgets? Is it time to ditch the cloud bill and move workloads back to on-prem? Of course, we’ve already spoken above about the cost of virtualization licenses.

Time equals money, and it’s not just developer productivity that’s seeing scrutiny: ops productivity will also be under the spotlight. For one thing, there’s unlikely to be budget for headcount, so if you’re feeling the lack of specialist K8s expertise, you’re unlikely to be able to hire it in. Expect those open reqs to stay unfilled.

And if you’re struggling to manage day-to-day operational tasks caring for clusters as Kubernetes use grows, or respond to new requests from internal customers, you may need to think about how you can do things smarter. CIOs will be comparing notes on the best in class for automation and if your platform engineers, sysadmins, architects and QA are soaking expensive hours in manual work across your clusters, it’s time to prepare for change.

5: It’s Not about the Distro

We’ve been saying it for a couple of years now, but the distro is just a layer in your stack, a component that you can swap in and out as your needs change (whether it’s for supporting limited hardware at the edge, or to get your workloads on to a different cloud).

It’s not that a distribution is a commodity: there are plenty of pros and cons and the choice matters. But you shouldn’t be committed to, or heaven forbid locked into, the first distro you deploy within production. You shouldn’t feel bad for using multiple distros (our research found that everybody does it). This is now mainstream behavior, just like multicloud — and just like multicloud, it has plenty of advantages, including risk mitigation.

The question then has changed. It’s no longer “what’s the best Kubernetes distro”, it’s “how do I manage multiple distros (and everything on top) without losing my mind.” The answer may surprise you.

Reasons to Be Cheerful

Some of these five trends might sound pessimistic, even alarmist. But they’re really not. They’re about Kubernetes coming of age.

We’ve passed the tipping point of Kubernetes being an exciting tech project: now we’re into enterprise technology considerations. Skills and team productivity. Choice and lock-in. Security and availability. And wrapping them all up, the most important of them all, cost-efficiency and return on investment.

As a community of technology pioneers meeting at KubeCon, we love discussing what’s cool and innovative. But we also need to account for a sustainable future for our ecosystem as Kubernetes hits the mainstream in a time of uncertainty.

Come and see us at Edge Day, or all week at KubeCon on booth S22, and share what’s on your mind.

The post 5 Trends to Watch for at KubeCon EU 2023 appeared first on The New Stack.

Write once, run anywhere. This mantra continues to hold true for the promise of WebAssembly (WASM) — but the keyword is “promise” since we are not there yet, especially for edge applications, or at least not completely. Of course, strides have been made as far as WebAssembly’s ability to accommodate different languages beyond JavaScript and Rust as vendors begin to support different languages, such as TypeScript, Python or C#.

As of today, WASM is very much present in the browser. It is also rapidly being used for backend server applications. And yet, much work needs to be done as far as getting to the stage where applications can reach the edge. The developer probably does not care that much — they just want their applications to run well and security wherever they are accessed, without wondering so much about why edge is not ready yet but when it will be.

Indeed, the developer might want to design one app deployed through a WebAssembly module that will be distributed across a wide variety of edge devices. Unlike years past when designing an application for a particular device could require a significant amount of time to reinvent the wheel for each device type, one of the beautiful things about WASM — once standardization is in place — is for the developer to create a voice-transcription application that can run not only on a smartphone or PC but in a minuscule edge device that can be hidden in a secret agent’s clothing during a mission. In other words, the application is deployed anywhere and everywhere across different edge environments simultaneously and seamlessly.

During the WASM I/O conference held in Barcelona, a few of the talks discussed successes for reaching the edge and other things that need to be accomplished before that will happen, namely, having standardized components in place for edge devices.

The Missing Link

What is an edge component? Microsoft’s Francisco Cabrera gave his description as he began his talk on AKS Edge Essentials and Akri on WASMs (and other tools). @wasm_io #wasm #wasmio23 @thenewstack https://t.co/aLf3IlarIk pic.twitter.com/o26VvMPKRS

— BC Gain (@bcamerongain) March 24, 2023

Edge is one of those buzzwords that can be misused or even misunderstood. For telcos, it might mean servers or different phone devices. Industrial devices might include IoT devices, applicable to any industry or consumer use for users that require connected devices with CPUs.

An organization might want to deploy WASM modules through a Kubernetes cluster to deploy and manage applications on edge devices. Such a WASM use case was the subject of the conference talk and demo “Connecting to devices at the edge with minimal footprint using AKS Edge Essentials and Akri on WASMs” given by Francisco Cabrera Lieutier, technical program manager for Micrsosoft, and virtually by Yu Jin Kim, product manager at Microsoft’s Edge and Platforms.

Edge deployments with WebAssembly were shown to work: AKS Edge Essentials and Akri on WASMs (and other tools) in this demo given by Microsoft’s Francisco Cabrera and Yu Jin Kim (virtually). Cameras capture apples! @wasm_io #wasm #wasmio23 @thenewstack https://t.co/BCpuUu87IR pic.twitter.com/rGlDDZy1o3

— BC Gain (@bcamerongain) March 24, 2023

Lieutier and Kim showed how a WASM module was used to deploy and manage camera devices through a Kubernetes environment. This was accomplished with AKS Edge Essentials and Akr. One of the main benefits of WASM’s low power was being able to manage the camera device remotely that like other edge devices, such as thermometers or other sensor types, would lack the CPU power to run Kubernetes that would otherwise be a requirement without WASM.

“How can we coordinate and manage these devices from the cluster?” Kim said.  The solution used in the demo is Akri, which is a Kubernetes features interface to makes connections to the IoT devices with WASM, Kim explained.

However, while different edge devices can be connected and managed with WASM with AKS Edge Essentials and Akri, the edge device network is not yet compatible with say an edge network running under an AWS cluster from the cloud or distributed directly from an on-premises environment.

Again, the issue is interoperability. “We know that WebAssembly already works. It does what you need to do and the feature set of WASM has already been proven in production, both in the browser and on the server,” Ralph Squillace, a principal program manager for Microsoft, Azure Core Upstream, told The New Stack during the conference sidelines.

“The thing that’s missing is we don’t have interoperability, which we call portability — the ability to take the same module and deploy it after rebuilding a different cloud but you need a common interface, common runtime experience and specialization. That’s what the component model provides for interoperability.

Not that progress is not being made, so hopefully, the interoperability issue will be solved and a standardized component model will be adopted for edge devices in the near future. As it stands now, WASI has emerged as the best candidate for extending the reach of Wasm beyond the browser. Described as a modular system interface for WebAssembly, it is proving apt to help solve the complexities of running Wasm runtimes anywhere there is a properly configured CPU — which has been one of the main selling points of WebAssembly since its creation. With standardization, the Wasi layers should eventually be able to run all different Wasm modules into components on any and all edge devices with a CPU.

@baihay showed w/@danologue the #Wasm Component Model with wasi-cloud for much needed standardization to run WebAssembly apps on cloud environments on @Azure. And I can confirm it really worked! @Cosmonic @bytecodeallies @wasm_io #wasmio23 @thenewstack https://t.co/aLf3IlarIk
. pic.twitter.com/19AUi4S1uY

— BC Gain (@bcamerongain) March 24, 2023

During the talk “wasi-cloud: The Future of Cloud Computing with WebAssembly,” Bailey Hayes, Bailey Hayes, director of the Bytecode Alliance Technical Standards Committee and a director at Cosmonic and Dan Chiarlone (virtually), an open source Software engineer at Microsoft’s WASM Container Upstream team, showed in a working demo how wasi-cloud offers standardized interfaces for running Wasm code on the cloud.

“Our answer to the question of how do you write one application that you can run anywhere across clouds is with wasi-cloud,” Hayes said. “And you can imagine that using standard APIs, one application is runnable anywhere or on any architecture, cloud or platform.”

The post What Wasm Needs to Reach the Edge appeared first on The New Stack.

The use of time series databases (TSDBs) has been prevalent in various industries for decades, particularly in finance and process-control systems. However, if you think you’ve been hearing more about them lately, you’re probably right.

The emergence of the Internet of Things (IoT) has led to a surge in the amount of time series data generated, prompting the need for purpose-built TSDBs.

Additionally, as IT infrastructure continues to expand, monitoring data from various sources such as servers, network devices, and microservices generates massive amounts of time series data, further highlighting the need for modern TSDBs.

While legacy time series solutions exist, many suffer from outdated architectures, limited scalability and integration challenges with modern data-analysis tools. Consequently, a new generation of time series databases has emerged, with over 20 new TSDBs being released in the past decade, particularly open source solutions.

These new TSDBs feature modern architectures that enable distributed processing and horizontal scaling, with open APIs that facilitate integration with data analysis tools and flexible deployment options in the cloud or on-premises.

Ensuring data quality and accuracy is critical for making data-driven decisions. Time series data can be “noisy” and contain missing or corrupted values. Time series databases often provide tools for cleaning and filtering data, as well as methods for detecting anomalies and outliers.

Here, we’ll explore the growing popularity of time series databases, the challenges associated with legacy solutions, and the features and benefits of modern TSDBs that make them an ideal choice for handling and analyzing vast amounts of time series data.

Kinetica is available as a fully managed service in the Cloud. Continued focus on core features and an improved Workbench interface drives growth among enterprises, new tech and government agencies including some of the largest IoT data analytics implementations in the world.

Learn More

The latest from Kinetica

What Is Time Series Data?

First, let’s take a step back. What is time series data?

Time series data is data that is characterized by its time-based nature, where each datapoint is associated with a timestamp that indicates when it was recorded. This is in contrast to other types of data (such as transactional or document-based), which may not have a timestamp or may not be organized in a time-based sequence.

As mentioned earlier, one of the main reasons why time series data is becoming more prevalent is the rise of IoT devices and other sources of streaming data. IoT devices, such as sensors and connected devices, often generate large volumes of time-stamped data that need to be collected and analyzed in real time.

For example, a smart building might collect data on temperature, humidity and occupancy, while a manufacturing plant might collect data on machine performance and product quality.

Similarly, cloud computing and big data technologies have made it easier to store and process large volumes of time series data, enabling organizations to extract insights and make data-driven decisions more quickly.

Another factor contributing to the prevalence of time series data is the increasing use of machine learning and artificial intelligence, which often rely on time series data to make predictions and detect anomalies.

For example, a machine learning model might be trained on time series data from a sensor network to predict when equipment is likely to fail or to detect when environmental conditions are outside of normal ranges.

How Is Time Series Data Used?

There are many applications and industries that commonly use time series data, including:

IoT and smart buildings. Smart buildings use sensors to collect time series data on environmental factors such as temperature, humidity and occupancy, as well as energy usage and equipment performance. This data is used to optimize building operations, reduce energy costs and improve occupant comfort.

Manufacturing. Manufacturing plants collect time series data on machine performance, product quality and supply chain logistics to improve efficiency, reduce waste and ensure quality control.

Retail. Retailers use time series data to track sales, inventory levels and customer behavior. This data is used to optimize pricing, inventory management and marketing strategies.

Telecommunications. Telecommunications companies use time series data to monitor network performance, identify issues and optimize capacity. This data includes information on call volume, network traffic and equipment performance.

Finance. Financial trading firms use time series data to track stock prices, trading volumes and other financial metrics. This data is used to make investment decisions, monitor market trends and develop trading algorithms.

Health care. Health care providers use time series data to monitor patient vitals, track disease outbreaks and identify trends in patient outcomes. This data is used to develop treatment plans, improve patient care and support public health initiatives.

Transportation and logistics. Transportation and logistics companies use time series data to track vehicle and shipment locations, monitor supply chain performance and optimize delivery routes.

Energy. The energy industry relies on time series data to monitor and optimize power generation, transmission and distribution. This data includes information on energy consumption, weather patterns and equipment performance.

In each of these industries, time series data is critical for real-time monitoring and decision-making. It allows organizations to detect anomalies, optimize processes and make data-driven decisions that improve efficiency, reduce costs and increase revenue.

Time Series vs. Traditional Databases

Applications that collect and analyze time series data face specific challenges that are not encountered in traditional data storage and analytics.

Time series databases offer several advantages over traditional databases and other storage solutions when it comes to handling time series data. Here are some of the key advantages:

High Write Throughput

Time series databases are designed to handle large volumes of incoming data points that arrive at a high frequency. They are optimized for high write throughput, which means they can ingest and store large amounts of data quickly and efficiently, in real time. This is essential for applications that generate a lot of streaming data, such as IoT devices or log files.

For example, an IoT application that collects sensor data from thousands of devices every second requires a TSDB that can handle a high volume of writes and provide high availability to ensure data is not lost.

Traditional databases, on the other hand, are designed for transactional workloads and are not optimized for write-intensive workloads.

Automatic Downsampling and Compression

Time series data is high-dimensional — meaning it has multiple attributes or dimensions, such as time, location, and sensor values. It can quickly consume large amounts of storage space. Storing and retrieving this data efficiently can require a significant amount of disk space and computational resources.

TSDBs often use downsampling and compression techniques to reduce storage requirements, providing more efficient storage without sacrificing accuracy. This makes it more cost-effective to store large volumes of time series data.

Downsampling involves aggregating multiple data points into a single data point at a coarser granularity, while compression involves reducing the size of the data by removing redundant information. These techniques help to reduce storage costs and improve query performance, as less data needs to be queried.

Specialized Query Languages

TSDBs often offer specialized query languages that are optimized for time series data.

These languages, such as InfluxQL, support common time-based operations such as windowing, filtering, and aggregation, and can perform these operations efficiently on large datasets. This makes it easier to extract insights from time series data and perform real-time analytics.

Time-Based Indexing

Time series databases use time-based indexing to organize data based on timestamps, which allows for fast and efficient retrieval of data based on time ranges. This is important for applications that need to respond to events as they happen.

For instance: A financial trading firm needs to be able to respond to changes in the market quickly to take advantage of opportunities. This requires a time series database that can provide fast query response times and support real-time analytics.

Traditional databases may not have this level of indexing, which can make querying time-based data slower and less efficient.

Data Retention Policies

Time series databases offer features for managing data retention policies, such as automatically deleting or archiving old data after a certain period. This is important for managing storage costs and ensuring that data is kept only for as long as it is necessary.

Scalability and High Availability

Scalability and high availability are critical for handling large volumes of time series data. TSDBs are designed to scale horizontally across multiple nodes, which enables users to handle increasing data volumes and user demands. Along with automatic load balancing and failover, these features are critical for supporting high-performance, real-time analytics on time series data.

TSDBs can also provide high availability, which ensures that data is always accessible and not lost due to hardware failures or other issues.

5 Popular Open Source Time Series Databases

There are several open source TSDBs available in the market today. These databases offer flexibility, scalability, and cost-effectiveness, making them an attractive option for organizations looking to handle and analyze vast amounts of time series data.

These open source TSDBs are constantly evolving and improving, and each has its own strengths and weaknesses. Choosing the right one depends on the specific use case, data requirements and your organization’s IT infrastructure.

Here are five popular open source TSDBs:

Prometheus

Prometheus is an open source monitoring system with a time series database for storing and querying metrics data. It is designed to work well with other cloud native technologies, such as Kubernetes and Docker, and features a powerful query language called PromQL.

InfluxDB

InfluxDB’s unified architecture, which combines APIs for storing and querying data, background processing for extract, transform and load (ETL) and monitoring, user dashboards and data visualization, makes it a powerful and versatile tool for handling time series data.

With Kapacitor for background processing and Chronograf for UI, InfluxDB, intially developed by InfluxData, offers a comprehensive solution that can meet the needs of organizations with complex data workflows.

Additionally, InfluxDB’s open source nature allows for customization and integration with other tools, making it a flexible solution that can adapt to changing business needs.

TDengine

This open source TSDB is designed to handle large-scale time series data generated by IoT devices and industrial applications, including connected cars. Its architecture is optimized for real-time data ingestion, processing and monitoring, allowing it to efficiently handle data sets of terabyte and even petabyte-scale per day.

TDengine’s cloud native architecture makes it highly scalable and flexible, making it an appropriate choice for organizations that require a high-performance, reliable time series database for their IoT applications.

TimescaleDB

A powerful open source database designed to handle time series data at scale while making SQL scalable. TimescaleDB builds on PostgreSQL, and it is packaged as a PostgreSQL extension, providing automatic partitioning across time and space based on partitioning key.

One of the key benefits of TimescaleDB is its fast ingest capabilities, which make it well-suited for the ingestion of large volumes of time series data. Additionally, TimescaleDB supports complex queries, making it easier to perform advanced data analysis.

QuestDB

QuestDB is an open source TSDB that provides high throughput ingestion and fast SQL queries with operational simplicity. QuestDB supports schema-agnostic ingestion, which means that it can handle data from multiple sources, including the InfluxDB line protocol, PostgreSQL wire protocol, and a REST API for bulk imports and exports.

One of the key benefits of QuestDB is its performance. It is designed to handle large volumes of time series data with low latency and high throughput, making it well-suited for use cases such as financial market data, application metrics, sensor data, real-time analytics, dashboards, and infrastructure monitoring.

Conclusion

Selecting an appropriate time series database for a specific use case requires careful consideration of the data types supported by the database. While some use cases may only require numeric data types, many other scenarios, such as IoT and connected cars, may require Boolean, integer, and string data types, among others.

Choosing a database that supports multiple data types can provide benefits such as enhanced compression ratios and reduced storage costs. However, it is important to also consider the additional complexity and costs associated with using such a database, including data validation, transformation and normalization.

Ultimately, the right TSDB can enable real-time data-driven insights and improve decision-making, while the wrong one can limit an organization’s ability to extract value from its data. As time marches on and the volume of time series data continues to grow, selecting a TSDB that is suited to your organization’s needs will become increasingly crucial.

The post What Are Time Series Databases, and Why Do You Need Them? appeared first on The New Stack.

Denis Lussier and Phillip Merrick, who worked together creating EnterpriseDB starting in 2005, are debuting a new project called pgEdge to address some of the toughest database challenges organizations face like data latency ultra-high availability and data residency restrictions.

With EnterpriseDB, which they call Door No. 1, they sought to add a Oracle compatibility layer on top of open source PostgreSQL. They call their latest venture Door No. 2, bringing a distributed Postgres database closer to the edge.

What Is the Edge?

Companies like Cloudflare, Akamai and Fastly have been handling compute closer to the edge, but Glauber Costa, founder and CEO of ChiselStrike, in a post called “What the Heck is Edge?” points out that bringing compute to the edge only solves half the problem if you’re making calls to a centralized database somewhere far away.

While there are many facets to the edge, including IoT devices, 5G and smart devices, edge development platforms like CloudFlare Workers, Fastly, Netlify or Vercel are focused on making more traditional web applications and enterprise applications faster by putting more of the application components closer to the edge. So pgEdge is bringing the database closer as well.

“For the web application developer who’s trying to shave milliseconds off the performance of the frontend of their application, if you’ve got a round trip to the database, which is probably someplace like Amazon us-east-1 that’s going to take you 100 milliseconds each time you need to access the database, then whatever work you’re doing on the frontend to optimize things is going to get swamped by that data latency problem,” Merrick explained.

Cockroach Labs senior software engineer (and TNS contributor) Paul Scanlon has been tracking the distance data has to travel in real-life edge scenarios.

The solution at pgEdge is reducing that distance with a distributed Postgres database.

“You’re not going to place 400 Postgres database nodes around the network. That would be a little excessive, and also unnecessary. So instead, you’re going to strategically place a handful of pgEdge nodes around the network, and each one of those nodes can take read or write traffic. And if you’re going to be using them in conjunction with edge platforms, like say Cloudflare Workers or Fastly, which is what we’ve designed this for, you’re going to be able to get really great response times out of the database layer,” Merrick said.

Kinetica is available as a fully managed service in the Cloud. Continued focus on core features and an improved Workbench interface drives growth among enterprises, new tech and government agencies including some of the largest IoT data analytics implementations in the world.

Learn More

The latest from Kinetica

Pure Postgres and Its Standard Extension

Having learned from their experience with EnterpriseDB, they wanted to start with a strong foundation layer. More than 35 years old, Postgres has been undergoing a resurgence lately.

“We haven’t rewritten the database engine underneath. It is Postgres. It’s not compatible with Postgres. It is Postgres, and that’s an important distinction,” Merrick said.

It also uses the Postgres standard extension mechanism. Its extension is called Spock, a reference to the logical Star Trek character. It’s a derivative of pgLogical that provides bidirectional multi-active (also known as multi-master) logical replication.

Nodes in a pgEdge cluster can span multiple cloud regions and data centers for high-level resiliency, availability and performance.

What that means for developers, especially those using the JAMStack architecture, according to Merrick:

“Yes, you can measure the page load times, but it’s also a less costly and easier route to getting very highly available database applications. Because if you’ve got multiple read/write nodes around the network, you can lose some number of them, depending on how many you have, and everything will just keep going independently. The traffic can fail over to the surviving nodes. So, so it’s a low-latency, ultra-high availability, distributed database.”

You can see a demo here.

Each node runs standard PostgreSQL (v15) and is kept updated via asynchronous logical replication with configurable conflict resolution and avoidance. It offers delta conflict avoidance to maintain consistency for data fields that keep running sums like financial balances or inventory counts.

It also offers geo-sharding to help users comply with regulations like GDPR that require certain data to remain within particular geographic boundaries. It breaks out tables on a location field so some data can be kept locally and while the rest is shared globally.

Existing Postgres Applications, Few if Any Changes

The company emerged from stealth earlier this month and announced a $9 million seed funding round. The technology is still in beta, with a handful of customers nearing production with it. The company is looking at Q3 for general availability.

“For leading e-commerce companies like ours, pgEdge is a game changer and will give us fast page loads and a smooth customer experience regardless of where our customer is located,” said David Ting, CTO of Zenni Optical, adding that pgEdge can also “give our engineers a solid foundation for building a new generation of e-commerce applications by allowing the developers to develop on native Postgres yet have the advantages of a modern distributed data store.”

Postgres users should be able to use their existing Postgres applications with few to no changes, Merrick said.

“ …you get to use all the tooling that works with Postgres, and many of the extensions that work with Postgres. So, [developers can] use it in exactly the fashion that they’re using Postgres today, whatever way their applications access it, whatever tools they like to use, for managing schemas and things like that. That’ll just work because it is standard Postgres.”

Bill Mitchell, CTO of PublicRelay, a SaaS media analytics company said, “We have been using pgEdge for a few months now, and I am very impressed with its performance and stability. Having their multi-master replication feature has allowed me to easily manage and scale my PostgreSQL databases without any issues. The product is easy to manage and that has saved me a lot of time and effort.”

All Open Source

Others also are trying to move the database closer to the edge. Costa’s company ChiselStrike, for instance, is attacking the latency problem with Turso, SQLite-compatible embedded database. EnterpriseDB also has a product similar to pgEdge called EDB Postgres Distributed (PGD).

Rivals like Cockroach, Yugabyte and others, in approaching the distributed edge problem, made the design decision to use a different database, then put a compatibility layer on top, according to Merrick.

“And we know from our work with Oracle compatibility, that will only get you 90% of the way there, and that last 10 percent is a bit of a killer,” he said.

A second differentiator is that the source code for pgEdge is all open source and available on GitHub.

The company has made its entire PostgreSQL distribution, a self-hosted and self-managed version called pgEdgePlatform that’s free and available for download. It also offers a managed database as a service called pgEdge Cloud on AWS and Azure. You can join the private beta here.

The post Startup pgEdge Tackles the Distributed Edge with Postgres appeared first on The New Stack.

Work on the first major revision of Svelte is underway, according to Rich Harris, creator of the frontend framework. This comes on the heels of the team’s release of SvelteKit, a full stack framework for building web applications.

“Now that SvelteKit is out, [is] pretty widely used and stable at this point, our attention is going to switch back to Svelte itself,” Harris told The New Stack. “Right now we’re working on Svelte 4, which is going to modernize the codebase.”

The team is switching the underlying code from TypeScript to JavaScript. That and the update will then allow the team to incorporate “big ideas” for Svelte 5 later this year, he added.

Svelte is a framework, but it’s also a language for describing user interfaces, Harris said. It’s compiled into JavaScript so that it can run anywhere. The advantage of doing that, he said, is when the application is built, Svelte knows which parts of the app could change and which parts can’t.

“Whereas [the first] frameworks would have to do a lot of work to figure out what needs to change on the page, Svelte was the first framework that really showed that that was unnecessary and that you could get significant gains, in terms of performance and in terms of bundle size, by doing as much work as possible ahead of time instead of in the browser,” Harris said.

What SvelteKit Offers Frontend Developers

Focusing on Svelte means there will be less focus on SvelteKit in the short term. SvelteKit was released as version 1.0 in December and is now in version 1.11. Harris said that it’s stable. It competes with Next.js, Gatsby and Nuxt.

“Sometimes people will say, ‘Should I start with Svelte or SvelteKit,’ as though they’re mutually exclusive. The way that I would frame it is, if you’re familiar with frameworks like React and Meta frameworks, Next or Remix, then Svelte and SvelteKit have a similar relationship,” Harris said.

SvelteKit is a user interface framework used to create self-contained components, which combine some markup, behaviors and styles into a reusable component that developers can use inside their apps, he said. That could be a navbar, a blog post or chat widget, or even a component inside another component, he added. It can be anything the developer wants.

“Increasingly over the last few years, what we’ve seen is framework teams realizing that it’s on us to provide tools for people to actually build applications with these component frameworks,” Harris said. “Next was probably the first really serious attempt at this. It took React from being something that you would have to cobble together into your own application framework.”

In short, if developers are building with Svelte, SvelteKit supports and provides the best way to do that, he explained. Svelte can run in two different environments — on the server or in the browser, where it will manipulate the DOM. SvelteKit is built with JavaScript and has the idea of the server/client split, Harris said.

“One is a one-shot, generate HTML, and you’re done,” he said. “The other is you’re creating this long-lived, potentially interactive thing, which might receive new data, and you can click buttons and create events and change state and all of those things, so it has to have this long life cycle.”

Even though you’re writing the component once, you’re targeting two very different environments, he added.

“SvelteKit gives you a really easy way to bridge that gap,” he said. “It will coordinate the initial rendering on the server, and then it will deliver a seamless handoff to the browser.”

The Benefits of SvelteKit’s Approach

The benefit of this approach is a better-perceived performance, because part of the page runs even as the JavaScript loads — or even if it won’t load, which Harris said happens more than developers tend to think. For instance, he experiences this problem whenever he’s on a subway — the connection drops out before the JavaScript can load. Having the server-side rendering can allow users to see the content anyway.

“It’s better for search engine optimization, it’s better for archival purposes, it’s better for accessibility, all of these other things,” he said. “That’s why we have this server/client mindset where both things are equal partners in the application.”

It does more than server side-rendering, though. SvelteKit also has a process for getting data from the server. If a page needs to update without reloading, it can fetch data from the server as well, enabling developers to create API endpoints so that data is available even to third parties within the same application, he said.

Edge Rendering and Svelte

Edge rendering is another type of server-side rendering. Using Svelte at the edge has generated some discussion and was a topic at the Svelte Summit two years ago. Harris said the ability is a function of the frontend frameworks, and not unique to Svelte.

“The big innovation with Edge functions is the not running Node anymore; they’re running a much lighter weight JavaScript runtime, which is practical to run in many different data centers all around the world,” Harris said. “The evolution we’re seeing is from centralized, manually managed servers to these very small units of compute that can run anywhere around the world. That can be any computer whatsoever, but in our case, it happens to be rendering HTML.”

Is Svelte Enterprise-Ready?

Harris readily acknowledges that, previously, Svelte might not have been the best choice for a large company.

“If you are responsible for engineering decisions at a large company, then the sorts of things that you’d be thinking about are: Does this framework have the backing of a major company? Are there a lot of developers using it? And for a long time, the answer was no and no,” he said.

That’s changed with the backing of Vercel, some of whose clients are now using Svelte. Schneider Electric, the Pudding and GitBook are among the companies that use Svelte.

“Increasingly, we are seeing developers at large companies use it,” Harris said. “Once you have one example of the company succeeding with Svelte that you know, the other companies will often follow.”

The post Rich Harris Talks SvelteKit and What’s Next for Svelte appeared first on The New Stack.

In recent months I’ve been taking a closer look at how data transfer works between the browser, the server, and the database, and have built a prototype application using Next.js and deployed it to Vercel to help visualize and calculate the distances that data has to travel between them.

I’ve been asking folks on Twitter to “submit their edge” so I can build up a better picture of where in the world requests to and from my databases are coming from. If you like, please go ahead and anonymously submit your edge.

You can preview the app and see the code on GitHub using the links below.

• 🚀 Live Preview: https://cockroachdb-edge-locations.vercel.app/
• ⚙️ GitHub Repository: https://github.com/PaulieScanlon/cockroachdb-edge-locations

Here’s a short explanation of what the dots on the globe represent.

• green dots: The approximate locations of folks who have submitted their edge.
• red dot: The location of a single region Vercel Serverless Function.
• orange dots: The locations of x3 multiregion AWS Lambda Functions.
• blue dots: The locations of x3 CockroachDB multiregion Serverless Databases.

Edge Explained

The green dots you’ll see around the globe are the approximate locations of users who have clicked the “submit” button. In the app I’m using request-ip and fastgeo-ip to translate the ip address on the req object of the function into real geographical locations. Then, using node-postgres I create a new row in the nearest CockroachDB multiregion database. CockroachDB handles replication across the other databases so that all three regions will stay in sync. (Full disclosure: I work for Cockroach Labs.)

You can see the src code for the “create” functions on the links below:

• ⚙️ Vercel Serverless Function: cockroachdb-edge-locations/pages/api/create.js
• ⚙️ AWS Lambda Function: edge-locations-serverless-api/create.js

Using the Vercel docs alone, I wasn’t able to determine exactly what these locations (green dots) were. For instance, when I submit, the location displayed (Horsham, West Sussex) is ~28 miles from where I actually am.

To get a clearer understanding, I asked Vercel directly. They were kind enough to reply to my email and here’s what they said:

My understanding of this is that we use MaxMind as a database to cross-reference the IP information and get a latitude/longitude for the request and then we output that. Maybe the best way to describe this is just an estimate of a user’s location based off of IP information using geolocation services accessed via Vercel — Vercel Sales Engineer

Now that I have a better understanding of the starting point, I can begin to plot the data journey.

Kinetica is available as a fully managed service in the Cloud. Continued focus on core features and an improved Workbench interface drives growth among enterprises, new tech and government agencies including some of the largest IoT data analytics implementations in the world.

Learn More

The latest from Kinetica

The Anatomy of a Request

As you may already know, you (your browser) can’t (in most cases) directly communicate with a database.

There are a number of reasons, many related to security (browsers aren’t secure). Instead, any requests you make from your browser have to go via a server; or, in this case, a Serverless Function or Lambda Function. The server can then perform the necessary security checks before requesting access to the database.

If all is “A-Okay!”, the database can respond in accordance with the request, e.g. with some data.

This leads me to the next part of the data journey. Where are the servers?

Where Are the Servers?

When deploying an app to Vercel, the static parts — HTML, CSS, Images — are deployed and globally distributed around a Content Delivery Network (CDN).

The idea being, the closer these assets are to the user, the shorter the distance traveled and the faster the website will “load”.

This, however, doesn’t apply to JavaScript executables.

A Serverless Function or Lambda Function can’t be deployed globally around a CDN; and instead, will be deployed to a region. Vercel is built on AWS and Vercel Serverless Functions can only be deployed to a subset of the available AWS regions.

Depending on the plan you’re using (I’m using a Hobby plan, but the same applies to the Pro plan), you can only deploy a Serverless Function to a single region. It is possible to deploy to multiple regions, but you’d need to upgrade to a Vercel Enterprise plan — Yikes!

For my purposes, I didn’t want to upgrade to a Vercel Enterprise plan, so instead I built a Route 53 API Gateway using AWS and then deployed multiple Lambda Functions to match the regions of the CockroachDB multiregion serverless databases, which are free and come with a 5GB storage limit. CockroachDB is cloud native and can be deployed to AWS or GCP, for this application I’ve deployed to AWS. This setup is currently costing me $2.30 a month.

Because I work at Cockroach Labs, I’ve been granted access to a private beta version of multiregion serverless (it will be available publicly later this year), which allows me to deploy my database to multiple regions. Once again the idea being, the shorter the distance traveled, the faster the website will “load”.

Vercel Single Region Serverless Function

With the limitations of the single region Serverless Function on Vercel (Hobby and Pro plans), no matter where in the world you are (including being geographically close to a database region), your request would have to go via the region of the Serverless Function.

Screenshot of Data journey using a Vercel Single Region Serverless Function

From my location in the UK, the request has to travel across the Atlantic Ocean to the Vercel Serverless Function located somewhere in us-east-1. From here my specific CockroachDB configuration optimizes the request and determines that the closest database is also located in us-east-1, accepts the request and sends the data back to the Serverless Function, which returns back across the Atlantic Ocean to my location in the UK.

The approximate distance traveled for this journey is as follows:

• One Way:
  • ~3,683 miles
  • ~5,928 kilometers
• Round Trip:
  •  ~7,367 miles
  • ~11,857 kilometers

AWS Multiregion Lambda Functions

With AWS multiregion Lambda Functions and the geographically aware API Gateway, requests are routed via the nearest Lambda Function.

Screenshot of Data journey using AWS Multiregion Lambda Functions

This time, from my location in the UK, the request only has to travel to eu-central-1 (Frankfurt) before, once again, my specific CockroachDB configuration optimizes the request and determines that the nearest database is also in eu-central-1. It accepts the request and sends the data back to the Lambda Function, which returns back to my location in the UK.

The approximate distance traveled for this journey is as follows:

• One Way:
  • ~452 miles
  • ~728 kilometers
• Round Trip:
  •  ~905 miles
  • ~1,456 kilometers

This results in the AWS multiregion route being ~87% shorter than the Vercel single region route.

If you’d like to try this yourself, hit “submit” and then use the toggle switch to submit via either the Vercel Serverless Function, or the AWS Lambda Function, and have a look at the distance of your data journey.

Infrastructure Management

I’ll be honest, setting up the API Gateway and Lambda Functions on AWS was no walk in the park. It was my first time using AWS and it did take a fair amount of reading to determine which of the many AWS services I needed to use.

AWS API

Below are some details that might help steer you in the right direction if you’re thinking of doing something similar.

Serverless (Framework)

I used Serverless to create the Lambda Functions and deployed them to multiple regions using a GitHub Action. You can see the repo for my API on the link below:

• ⚙️ GitHub Repository: https://github.com/PaulieScanlon/edge-locations-serverless-api

API Gateway

The Lambda Functions are deployed behind an API Gateway with a custom domain. You can see the default route on the link below; depending on where you are in the world, you’ll see an appropriate region. For me, in the UK the region displayed is eu-central-1, but yours might be different.

• 🔗 GET /: https://api.crl-devrel.net/

Route 53

I’m using a Route 53 hosted zone. I’ve added three A Records with Geolocation routing and defined a Differentiator to route traffic from different regions to the three Lambda functions.

Screenshot of Route 53 Hosted Zone A Records

Vercel Infrastructure

The advantage of Vercel is there’s little to no infrastructure setup required. However, in my case it was more of a hindrance than a help. The application I’ve developed and the requirements I had didn’t quite fit the pricing model — curse my luck!

AWS Infrastructure

It’s not the easiest thing to do, but not the hardest either. I’d also like to add, don’t let the gatekeepers deter you from trying this yourself — managing infrastructure can provide a level of flexibility you might not achieve using a managed service like Vercel. If you’ve got the time and curiosity, it’s worth a look. It’s also worth noting, having AWS knowledge is a valuable skill to have.

Edge Functions

I’ve been publicly discussing this project on Twitter and I’ve had a number of folks mention I should use Edge Functions. I do need to dig a little deeper into this but for now I’ll just quote the Vercel docs verbatim: Most Node.js APIs are not available.

Screenshot of Edge Function Node.js Limitaitons

Edge Functions also have additional limitations relating to the code size limit. These are:

• Hobby – 1MB
• Pro – 2MB
• Enterprise 4MB

My requirement to use node-postgres would exceed the 4MB limit on a Vercel Enterprise plan, so for now at least, Edge Functions aren’t quite the right fit for this project.

If you find yourself in a similar situation I think it’s worth considering alternatives. AWS has a fantastic array of services and whilst it can be a bit tricky to get started, I personally feel (with my limited knowledge) there are fewer restrictions.

I plan to keep learning about AWS and building in public and if you’re doing the same, let’s talk! You can find me on Twitter here: @PaulieScanlon.

In the meantime, go ahead and submit and lemme see your Edge.

Further Reading

• 5 reasons to build multiregion application architecture
• Getting Started With CockroachDB, pg-promise and Next.js
• How we built easy row-level data homing in CockroachDB with REGIONAL BY ROW

The post The Distance from Data to You in Edge Computing appeared first on The New Stack.

If you’ve been interested in working on an edge computing or IoT project but didn’t know where to start, this article will help give you an overview of what options are available when it comes to the hardware to power your project.

Edge Computing Overview

What the “edge” actually is can be drastically different based on your frame of reference. The edge could be as simple as a content delivery network (CDN) serving static files to lower loading times, having your app deployed in multiple regional data centers, using modern serverless frameworks and services to move processing to hardware closer to users or even pushing features directly onto the user’s device.

The main reason to use edge computing comes down to a tradeoff between latency, cost and reliability. Latency frustrates users, so developers want to reduce it whenever possible. Bandwidth can also be expensive if you are using cellular or satellite data for devices in the field, so for data-heavy tasks, it makes sense to try and do things on the device when possible.

For critical applications where reliability is extremely important and network connectivity can’t be guaranteed, edge computing can ensure things keep functioning by doing the work locally rather than relying on processing power in the cloud.

Here are a few common examples of edge computing:

• Security camera image recognition
• Self-driving cars
• Industrial manufacturing anomaly detection
• Industrial monitoring

InfluxData is the creator of InfluxDB time series platform. InfluxDB empowers developers to build real-time IoT, analytics, and cloud applications with time-stamped data. It is purpose-built to handle the massive volumes of data produced by sensors, systems, or applications that change over time.

Learn More

The latest from InfluxData

Nvidia Jetson

The Jetson series of chips made by Nvidia is designed for edge-computing workloads that need the ability to perform artificial intelligence tasks on the edge rather than relying on data center GPUs. Example use cases are things like robotic arms with AI-powered vision, cameras that can perform streaming analytics and advanced machine-learning models running on collected sensor data.

Nvidia makes development easier by providing a variety of software tools to use with Jetson hardware to create a unified development stack. Jetson has development kits available for prototypes or hobby projects as well as the ability to buy chips in bulk if you plan on commercializing what you build.

Particle

For edge or IoT projects where WiFi or ethernet isn’t an option, the Particle series is a great choice. Particle provides globally available SIM cards, which allow your devices to transmit data using cellular data.

Another benefit of using Particle beyond the hardware is that Particle provides an entire IoT cloud platform that makes integrating different systems together much easier than trying to build it on your own. Particle also provides a number of APIs, software development kits (SDKs) and integrations with third-party tools.

Google Coral

If your project involves any type of machine learning, the Coral chips made by Google are a great option. All of the Coral devices are powered by Google’s custom tensor-processing units (TPUs), which are specialized for machine-learning workloads. Coral provides devices designed for fast prototyping and production-ready devices for situations where you want to commercialize what you created using a prototype board.

One really cool device made by Coral is the USB accelerator, which can be used in combination with some of the other devices talked about in this article to give them extra power to run machine-learning models efficiently.

Source

Raspberry Pi

The Raspberry Pi is probably the most well-known product when it comes to getting started with IoT or edge computing–type projects. The main selling point of Raspberry Pi is its usability for beginners, the ecosystem and the community. Almost any task you would want to attempt probably has a guide or tutorial available to help you out, and there are a number of libraries and third-party hardware to speed up development.

The only real downside about the Raspberry Pi in the last few years is that it has been almost impossible to find due to chip shortages. According to the company, this problem should be resolved in the second half of 2023.

Orange Pi

As the name suggests, Orange Pi devices have quite a bit of overlap with Raspberry Pi. The main benefit of Orange Pi is that you will get better hardware performance for the same or lower price, compared to Raspberry Pi.

The downside is that Orange Pi doesn’t have nearly as large of a community or third-party ecosystem. These problems can be mitigated by using certain Orange Pi device models that are made to be compatible with many of the peripherals designed for Raspberry Pi boards.

Odroid

Odroid is a series of single-board computers created by Hardkernal that are a good fit for edge computing. Odroids are considered similar to Raspberry Pis in terms of where it makes sense to use them. Some benefits of using an Odroid as opposed to a Raspberry Pi are that you can add external RAM to certain Odroid models and, in general, there are more options when it comes to processing power and configuration.

A typically cited downside is that Odroids are considered a bit more difficult to work with when it comes to setting things up and working with peripherals compared to Raspberry Pi’s user-friendly ecosystem.

Intel NUC

Intel NUCs are essentially mini-PCs that are an excellent option for edge computing due to their small-form factor. These devices pack a lot of computing power into a small package, making them ideal for space-constrained environments.

Source

They are designed to be easily extendable and upgradeable, with easy access to components like RAM and storage. NUCs can also be configured to run a variety of operating systems, including Windows and Linux. Another bonus NUCs have for edge computing is that they consume relatively little power, which can be important in situations where power usage is a concern. A potential downside is that they don’t tend to have as many ports or support for peripherals out of the box compared to some of the other devices in this article. Whether that is an issue will depend on your use case.

One example of Intel NUCs being used for edge computing at-scale is Chick-Fil-A, which has a series of popular blog posts about how they deploy Kubernetes at each of their stores using Intel NUCs for hardware.

Next Steps

Beyond the devices covered in this article, there is a wide range of other options available for IoT or edge computing, which all have their own tradeoffs. Which you choose will depend on the requirements for your particular project and how much you want to optimize for costs over convenience and development speed.

Check out these links for more information and tutorials that will help you with your IoT or edge computing projects:

• How to make an IoT monitoring application with Python
• What to look for in a database for storing IoT data
• Node Red beginner tutorial for IoT
• IoT beginner project ideas

The post 7 Hardware Devices for Edge Computing Projects in 2023 appeared first on The New Stack.

Eleventy is a static site generator built for speed, and the reason it can deliver is that it leaves a bundler behind, said Eleventy creator Zach Leatherman, in an interview with The New Stack.

“The real reason that Eleventy is, in many ways, a lot faster than a lot of the competitors we have in the same spaces, is that we don’t use a bundler,” Leatherman told The New Stack. “We’re not strictly tightly coupled to a bundler.”

Leatherman’s Case against Bundlers

A lot of site generators do use a bundler, he added. For example, Next.js used Webpack but is pivoting to Turbopack. Vite is another popular bundler, which Eleventy does support with a plugin.

“But the more work you do in your build, the slower it is, and that’s just kind of how it is,” Leatherman said. “We’re trying to bet on the post-bundler phase of development and the trend kind of oscillates back and forth between what you need a bundler to do and what you don’t need a bundler to do.”

Static site generators use HTML, CSS and JavaScript to create large websites, automating the page creation. Other site generators that compete with Eleventy include Astro, Gatsby, Next.js, Next, Remix and SvelteKit.

Leatherman has run tests that showed Eleventy version 1 ran faster than most competitors, building ×4000 markdown files in 1.93 seconds compared to 22.90 seconds for Astro, 29.05 seconds for Gatsby and 70.65 seconds for Next.js. Only Hugo, at 0.68 seconds, beat Eleventy.

As web standards improve over time, the need for features supported by a bundler decreases, he explained. One example: ECMA script modules. A few years ago, many bundlers were transforming code that was using ECMAScript modules to a more broadly accessible form of JavaScript. But as web standards evolved and browsers improved and adopted those things natively into the browser, it was no longer necessary for a bundler to support those features, he said. Often, “expensive” features go into a bundler to handle use cases that aren’t even relevant anymore, he added.

“My argument is that you don’t need a bundler,” he said. “For most web development use cases, a bundler is not necessary. Now, you can use a bundler with Eleventy if you’d like to integrate those two things together. But I’ve been building sites for years and years and years and, yeah, I would argue that the churn and the performance hit that you get from a bundler in many sites, and many use cases is not worth it.”

Eleventy uses Node.js as its runtime, so that must be installed to run Eleventy. Then it’s installed via npm, which is bundled with Node.js. Eleventy is run through the command line.

“We want to be one layer above just writing an HTML file in your editor,” Leatherman said. “Eleventy will allow you to automate some of the things that go into building a website.”

For instance, Eleventy will repeat a header or footer in a scripted way so that the developer doesn’t have to copy and paste the code.

Eleventy is sponsored by Netlify but owned by Leatherman. Version 2.0 was officially released on Feb. 8; it included only a few updates from the beta because Leatherman was already working with canary versions of the software over the past year.

“I’ve been using it in production for a very long time and working on it in the pre-release, canary versions really allowed me to do live iterations with production sites,” he said. “The thing that folks like about Eleventy is that it’s very stable. A lot of folks that have come back to projects that were using older versions of Eleventy to upgrade to Eleventy 2.0 have said that it was a very seamless upgrade, and they didn’t have a ton of development that needed to go into that upgrade process.

“Stability is kind of a hard thing to sell, but our track record of being a five or six-year-old project now and really delivering value to folks over those five or six years speaks for itself,” he added.

New Dev Server and Web Components

One update that did ship was a new dev server.

“That allowed us to get rid of a lot of dependency weight, so a lot on the size of Eleventy as people installed on their computer,” Leatherman said. “So Eleventy is much more lightweight when you install it. I think it’s something like 34 megabytes when you install it now instead of 150 megabytes.”

Another update that excites Leatherman is the new WebC file format, which stands for “web component” and includes a number of benefits, including support for streaming on the edge.

“The JavaScript ecosystem has really gone really hard into third-party or independent component frameworks. So your React, your Angular, those are all sort of things that exist independent of the platform, independent of web standards,” he said. “They exist a layer above and there has been a lot of tension between web components and the web standards folks, and the framework library folks as well. There’s been a fair amount of disagreement between those two teams as to what component frameworks should look like when it comes to being part of the platform and part of web standards.”

WebC is trying to bridge the gap between those two worlds, he explained.

“We can have as much long-term benefit from the platform without getting into the same churn problems or heavy JavaScript problems that we see from a lot of JavaScript frameworks today,” he said.

The post The Need for Speed: Why Eleventy Leaves Bundlers Behind appeared first on The New Stack.

Let’s face it: our industry has a penchant for buzzwords. As people try to differentiate themselves and establish their own relevance, they often try to do this by inventing and capturing terms that may or may not mean something.

I’m old enough to remember when people tried to push things like grid computing and fog computing.

So you would be forgiven for thinking that “the Edge” is just yet another such buzzword. Especially because the term’s confusion is made worse by the fact that:

• Two different slices of the industry mean two completely different things by “edge.”
• People often conflate it with serverless — which had its own misconceptions.

In this article, I’ll explain what the edge is and why you should care.

The Definition

Wikipedia defines edge Computing as “a distributed computing paradigm that brings computation and data storage closer to the sources of data. This is expected to improve response times and save bandwidth.”

We spot the reason for the first confusion in this very definition: closer to what? If it brings computation and data storage closer to the sources of data, that is a comparison. If I have a database in us-east-1 and replicate it to us-west-1, is that the “edge?”

The ‘Far Edge’ or ‘Offline Edge’

When we started talking about a product for the edge, an old colleague of mine immediately sent me a message, questioning that what we were doing was not really the edge. This person has been working with “the edge” for years, and when talking to him, the confusion immediately became clear.

Developers working on the Internet of Things (IoT), mobile devices, points of sales or telecom, see “the edge” as something as close as possible to the devices themselves, potentially inside the devices. When they talk about “pushing compute to the edge,” they are likely talking about compute that happens inside the device.

One immediate characteristic of those deployments is that internet connectivity is either slow, intermittent or barely existent (think airplanes, industrial controllers, etc). Compute and storage resources are not just reduced in comparison with the cloud, but they’re severely reduced.

Because of these characteristics, this can be referred to as the “far edge.” A term I personally prefer is “offline edge.” This is the domain where offline-first solutions shine.

The ‘Near Edge’ or ‘Online Edge’ (Web)

I could never convince my friend that there is “another edge;” to this day, he likely just thinks I have no idea what I am talking about. And in all fairness, as far as I know, the IoT folks got there first. So if they want to claim that theirs is the true edge, there’s very little I can do.

But the reality is that there is a growing and strong segment of the industry that uses the term edge to mean something else entirely. Those are web developers, deploying their applications to platforms like Cloudflare, Vercel or Netlify.

While “the cloud” may offer you a region “in Europe,” the edge will offer you multiple cities spread across Europe. Those edge locations are not as powerful and capable as cloud data centers, but they are still data centers. Technically they could be as powerful as any other cloud data center, but this wouldn’t be economical.

The web edge: lowering response times to applications, but still mostly online.

The Missing Piece: The Data Edge

For the IoT edge, each terminal device will bring its own local storage. But for the web edge, things are different. Bringing data to the edge presents its own challenges.

Compute and storage are both present in the definition of edge computing, but they are very different in nature. Compute is nimble and can be easily moved anywhere. Data is heavy and moving it has a cost. Compute is unencumbered by regulations and can happen anywhere. Data is protected and has to be treated differently depending on the jurisdiction.

For this reason, companies targeting the edge have so far focused most of their efforts on compute, like edge functions. But bringing compute to the edge only solves half the problem, especially if you’re making calls to a centralized database somewhere in a faraway cluster. You end up with as much, or potentially more, latency than if you had just hosted all your compute in a traditional centralized cloud-hosted location.

One way that developers move data to the edge is to cache it on end users’ devices. This works well for offline use, but caching and synchronization are hard things to get right, and it doesn’t solve many cases where fast access is required.

These data edge problems are exactly the ones that my company, ChiselStrike, aims to address with Turso, which brings the power of a SQLite-compatible embedded database to the edge. Turso is built on libSQL, an Open Contribution fork of SQLite.

Is Edge the Same as Serverless?

When companies like Vercel, Netlify and Cloudflare talk about the edge, they usually refer to “edge functions.” This leads developers to conflate “edge” and “serverless” as if they were similar or the same. With serverless, although there are servers somewhere, developers don’t think in terms of servers that need to be configured, scaled up or scaled down. They think in terms of functions that get executed — they are just deploying code and expecting the cloud provider to handle the rest.

This paradigm goes very well with edge computing: since we are assuming there are fewer resources available, allowing potentially wasteful general-purpose compute is less enticing. Constraining what the developers can do (through functions and massive multi-tenancy) allows for better utilization and resource packing.

Another reason is historical: those platforms were initially simple content delivery networks (CDNs), handling static assets. Over time, those CDNs started to increase in functionality and become programmable CDNs.

CDNs are gaining functionality, effectively evolving into the edge. The data edge is the next frontier.

But companies like Fly.io are also edge companies, and they operate on a different model. You can deploy a container abstraction with a long-lived application of your choice, and because they make that application available in many regions, they are also an edge player.

Is the Edge “Just X”?

Every time a new term comes along, people get too far in the weeds about whether something is truly “novel” or not. When the cloud was born, lots of people didn’t see the value because it was “just renting servers.” It was not: the ability to rent those servers in seconds instead of weeks and give them back just as easily changed how businesses operated.

I’ll always remember a coworker of mine that didn’t see any value to Slack because “it was just IRC,” and my favorite to this day is still this timeless Hackernews comment about Dropbox:

So is there something really transformative about the online edge? Yes!

Before the online edge, you could build applications that span multiple locations the same way you could have built elastic applications before the cloud: by being painfully precise about geographical provisioning, routing, placement and execution.

What the edge allows you to do now is to code as if none of that matters — as if the world is one — and to set policies about what is allowed to happen where. The edge allows truly global applications to be built transparently, without investing time and money in explicitly setting up and managing a global presence.

The Second Criterion for Relevance

The edge fits the first criterion of whether or not something is “just X” or a new concept that is here to stay: it does transform the way in which developers architect their applications. The second criterion is whether or not it solves a distinct business problem.

And this is a question for you, the reader: do you have a need to build applications that serve multiple locations with low response time? We built Turso on the strong belief that, for many of you, the answer is an unequivocal “yes.” If so, we would love to invite you to connect with me on Twitter and other edge enthusiasts in our Discord community.

The post What the Heck Is the Edge — and Why Should You Care? appeared first on The New Stack.

I was watching a developer give a demo recently, and part of it included a look at a dashboard of incoming requests and the response codes. I asked the developer how she was generating the request traffic. “Oh I’m not,” she replied. “I’ve just had a public IP up for more than 10 minutes, and people are hitting it looking for vulnerabilities.”

I’m used to protecting an active website but not a random IP address. That is the reality of today’s web development environment. If you have an API that operates over a network, you have to secure it because someone is going to attack it, guaranteed.

Ideally, you can immediately enact a zero-trust model, where even traffic inside your network is checked at all stages. But implementing systems, whether they are security, deployment, continuous integration or otherwise, is often done in steps. A great place to start with network security is at the edge of your network, where it interacts with users and attackers.

Often an API gateway will be the first part of your system to receive a request, making it a good tool to be your first line of defense. In general, in a cloud native architecture, we try to remove as much burden from application developers as possible by extracting functions like observability to sidecar services.

The same is true of authorization, and handling authorization in your API gateway is like having a sidecar for your larger system. There are two major domains of security an API gateway should be able to handle itself or have integrations for:

• Authentication (Authn)
• Authorization (Authz)

We’ll focus here on these topics as they relate to API gateways, but if you need a primer on either, Auth0 has great guides to both authentication and authorization.

Ambassador Labs is the cloud native developer experience leader. The company’s developer control plane for Kubernetes integrates the development, deployment, and production infrastructure for developers and organizations worldwide so they can code, ship and run apps faster and easier than ever. Ambassador and TNS are under common control.

Learn More

The latest from Ambassador Labs

Authentication and Authorization

In brief, a system uses authentication and authorization processes to answer two questions for a given request:

• Who made this request? (authentication)
• Are they allowed to do what they are trying to do? (authorization)

This is true whether the request comes from outside or inside the system, as in a request from another service or in an identity and access management (IAM) system. Some tools overlap both functions. For example, a JSON Web Token (JWT) can be used to identify a user and to contain a list of authorized permissions in its payload.

Authentication for APIs

The core of authentication is checking a user’s credentials against a known entity and sending back a credential of some kind for the user to send in subsequent requests, whether that’s a JWT, a session cookie, etc. What you are looking for in an API gateway is built-in compatibility with as many of the standard authentication methods as possible and extensibility, so that you can use other methods the gateway doesn’t have built-in.

Some common methods of authentication to look for in an API gateway are:

• Basic authentication — also known as “username and password.”
• OpenID Connect (OIDC) — this is the tool used for single sign-on (SSO) by most major providers like Azure Active Directory, Google, Github, Okta, etc.
• JWT — to receive a JWT, a user will need to have authenticated in some other way but will send the JWT for future requests for as long as the token is valid. Quickly validating the JWT at the point of entry is highly convenient.

Extensibility

It is common for large business systems to end up with a custom or uncommon authentication system for an older or highly bespoke service. You may not need to be able to extend your API gateway’s authentication system when you first start using it, but it’s always nice to know that you can. Typically, this will look like a feature allowing you to configure the API gateway to send a request out to an external service that you create and receive back a response approving or denying the request.

Authorization for APIs

Authorization is a trickier subject because it is often business- and domain-specific. Even when there are common types of authorization systems like role-based access control (RBAC), the roles and what each role can do would be specific to your system. That said, there are a few capabilities to look for in an API gateway that can be considered authorization, and you can look for the ability to integrate it with your systems, the same as with authentication.

Authorization Capabilities of API Gateways

If authorization is trying to answer the question, “What are you allowed to do?,” then there are some permissions at the edge of your network that your gateway should be able to validate:

• Are you allowed to make this request at all? Your API gateway should be capable of returning an immediate 403 or equivalent “Not authorized” response if a user attempts to make a request they should not.
• How fast are you allowed to make requests? Rate limiting is a form of permission and is within an API gateway’s wheelhouse. Ideally, you should be able to configure your gateway to rate limit globally, by IP address and user ID.
• To where should you be allowed to make requests? The systems that should handle your requests can be a part of authorization. For example, in a multiregional architecture, you might require that EU customers can only make requests to EU servers as Monday.com does. Your API gateway should be able to do dynamic routing to support that.
• Work with OAuth2, which is often found where you use OIDC. Where OIDC verifies your identity, the OAuth2 protocol provides access authorizations with the same systems.

Integration

The multiregional architecture example is the beginning of a larger pattern. Your API gateway should be able to send requests out to your authorization service and receive back instructions on what to do with them. Should they be forwarded? Redirected? Dropped? Because authorization tends to be bespoke, but you want to protect your network at its edge, your edge needs to be able to integrate with your bespoke systems.

Start at the Edge

API security is a requirement now when an attacker is always probing your network, looking for a way in. Without it, you are one bad day away from a breach and very unhappy customers.

Just as you might start with implementing TLS at the edge of your network with an API gateway and then add in mutual-TLS between internal services using a service mesh, you can implement security measures at the edge while working toward a full zero-trust system.

If you want to learn more, the latest edition of the online KubeCrash event, March 22 – 23 will dive deeper into the topic of zero trust, covering topics from the ingress to service mesh to certification generation and policy enforcement.

The post Security at the Edge: Authentication and Authorization for APIs appeared first on The New Stack.

Kubernetes makes sense at the edge if deployed with the right tooling.

At last week’s Civo Navigate conference, Portainer demoed how it simplifies edge Kubernetes deployments, using what it touts as a universal container management tool. Basically, it can work with both Docker and Kubernetes to make the deployment and management of containerized apps and services easier, Jack Wallen said in his assessment last year.

Kubernetes makes sense at the edge, Portainer co-founder Neil Cresswell told Civo audiences.

“One of the major benefits with Kubernetes is that really we have a recipe or a means to declare how we want our application to run and that’s the manifest,” Cresswell said. “The manifest really is a game changer. And in my career, it’s the first time I’ve ever seen a way that you have a way to basically declare how your application runs, and it runs that way everywhere.”

While it can be overwhelming, the manifest makes sense once you’ve wrapped your head around how it works — that it’s infinitely reproducible, he explained.

“No matter where you run the manifests it’ll pretty much run the same way every single time,” he said, adding Portainer ships its product as a Kubernetes manifest. Using Portainer to run a manifest means it will run in seconds and in very predictable ways, and it will deploy the same way, he said.

For the purposes of the demo, Cresswell defined the edge as the network’s edge.

“When we say network edge what we’re talking about here is putting applications closer to users,” he said. “We do this to try to reduce network latency and reduce bandwidth. But the whole thing there is to say how do I get an application close to my user so they get a snappy responsive application experience and I pay less for bandwidth between my user and the backend system.”

He gave an example of why this matters today more than ever — he has a bank app that, when he travels to the U.S., experiences excess latency issues because it’s trying to hit the backend of his bank in New Zealand.

“Whenever I am here [the U.S], my internet banking application is completely unusable because it is trying to do API requests — 1000s of them — to a backend server in New Zealand,” he said. “That latency just breaks the application. So by being able to have API endpoints closer to your users, you get a much faster application experience.”

Stateless Services and Kubernetes

The edge relies on stateless services and the Kubernetes working group has done a really good job helping to transition it to more stateful services, but predominantly Kubernetes is built around stateless, and edge applications are also predominantly stateless.

“They’re designed for ingesting and tuning and buffering and caching. They’re not designed really to hold data,” he said. “So Kubernetes is a stateless orchestrator and edge — stateless workloads — really are a perfect match. It just makes sense.”

Moving applications to edge makes it possible to reduce bandwidth between users and the backend; reduce latency overall; provide a faster app experience; and support data separation.

That said, there are challenges to deploying Kubernetes at the edge, he said. While certified Kubernetes distributions use a standardized API, a lot of providers want to add value to the native Kubernetes API, which leads to lock-in. That’s why it’s important to be careful about things like authentication and load balancing, he added, which can lock developers into a particular buyer.

Cresswell showed a diagram of providers, noting that Civo, Linode and Digital Ocean all provide the raw Kubernetes without adding aspects that will lock you in. Azure and Google also offer raw environments, though they do provide some value add but are generally quite compliant, he said. Other providers may make it more difficult, he added, to “have a fully transformed work location.”

Multiple Cluster Kubernetes Management

The Kubernetes management challenge comes in when you’re deploying multiple clusters rather than a single cluster, Cresswell said.

“When you get to three, four, 10 clusters, things start a little harder,” he said. “When you’re talking edge, though, you really are talking standard and you really have to change the way you think about how I manage this.”

Among the issues to consider are:

• How do you manage the clusters centrally?
• How do you perform authentication centrally?
• How will users automatically get propagated to backend clusters?
• How do you define back roles somewhere centrally and have those propagate?

There are three main features developers need, he said:

1. Centralized access;
2. Access control with monitoring dashboards; and
3. Centralized deployments.

“You really have to have all of those,” he said, and you have to do it at scale. While it’s possible to go cluster-by-cluster, you still have to think about user authentication.
“You really do have to say, ‘I want to have a single API endpoint that every developer or consumer can connect to, and their proxies, to a backend,” Cresswell said. “This is how you manage things at scale. And the same thing with dashboards.”

One Dashboard to Manage the Edge

Cresswell said that everyone thinks they can install Prometheus or Grafana and have a monitoring dashboard — but you don’t want to have 47 different dashboards open.

“You want to try and get a macro global view of where the clusters are. You can do that with Prometheus and Grafana, but you have to architect it that way,” he said. “You can’t just don’t install it in each cluster. Yes, you have to install the edge, the Prometheus edge agents, and send the streams back to a central Prometheus instance. You have to configure things correctly. So you have to think differently. And bulk deployments is actually quite complicated.”

GitOps will help but it won’t get you there all the way, he added, specifically pointing to a Chik-fil-A rollout of a couple 1000 clusters where they started using GitOps and “found out very early on that you can’t just deploy GitOps and pray that it’s going to update,” he said. “You actually have to have some centralized dashboard to see the current status of all these deployments.”

Portainer faced the same challenge, he said, and has added tooling to manage these clusters, along with identity management and access management, from a centralized dashboard, Cresswell said.

“Portainer has a high level of abstraction; we try and make things really easy,” he said “We try to be the one tool you need to manage Kubernetes… So full Kubernetes API proxy, multicluster management, centralized identity management, and dashboards, monitoring everything you need to basically go live with Kubernetes in production at scale.”

Cresswell and Adolfo Delorenzo, Portainer’s IT sales and business development executive, demoed how Portainer could manage the edge from one dashboard by broadcasting a live stream of the audience into three different locations supported by Civo — London, New York and Frankfurt — in about two minutes.

“We have customers who’ve got upwards of 50,000 environments. We have a customer who need us to stand us up 125,000 clusters by the end of this year,” Cresswell said.

Civo paid for Loraine Lawson’s travel and accommodations to attend the conference.

The post Portainer Shows How to Manage Kubernetes at the Edge appeared first on The New Stack.

Social media giant Pinterest improved Android video performance by optimizing warm-ups, configurations, and player pooling with a few simple steps, according to a recent blog post from company engineers.

Engineers of the site consider their work a “never-ending” investigation into the inner workings of ExoPlayer, their preferred application-level media player for Android.

Although geared to Android, the blog post — written by Grey Skold, former Pinterest Android video engineer, Lin Wang, Android performance engineer, and Sheng Liu, Android performance engineer — featured underlying concepts that are applicable in other areas.

Let’s take a look:

Warm up

The key learning here was to establish the network connection right away. Rather than waiting to return video URLs from the server, they establish a network connection during the application start-up time via a dummy HTTP HEAD request. This same connection is used to play future videos.

This same strategy is applied to UI rendering.

A similar, pre-establishing strategy is employed for UI rendering. ExoPlayer’s default is to parse the video URL to:

• calculate the aspect ratio
• invoke on onContentAspectRarioChanged()
• inform the AspectRatioFrameLayout of the aspect ratio.

Since most of the video’s aspect ratios are pre-determined, the following functions can prevent that work from happening:

• setting the video aspect ratio via AspectRatioFrameLayout.setAspectRatio()
• Override the PlayerView.onContentAspectRatioChanged() method with an empty body (this prevents the player from trying to recalculate the aspect ratio).

Configurations

Short Term Buffering

Buffering delays playback until enough data is buffered. Since most of Pinterest’s video is short form, the engineers opted for shorter buffering durations. The result is less time to wait and faster time to load. setBufferDurationMs() is the ExoPlayer function for setting buffering durations. The rebuffer rate increased but the overall video UX improved.

Limiting Size and Sound

The two parameters below load videos in-feed and limit the size to the correct viewport size. This eliminates any 1080p video sizes in 360-pixel viewports.

This set of parameters disables audio rendering, allowing Pinterest to play multiple muted videos at the same time. This saves network bandwidth from the download and memory consumption from processing.

Cleaning a Dirty Cache

ExoPlayer provides a cache interface that kept downloaded media files on disk. The catch though is that when Pinterest ran into fatal errors caused by backend bugs, the bad contents also stuck in the cache causing errors long after the bugs were fixed.

The solution is to use SimpleCache.removeResource() to purge the dirty cache when the following fatal IO errors are returned from the Player.Listener.onPlayerError():

• ERROR_CODE_IO_UNSPECIFIED 2000
• ERROR_CODE_IO_INVALID_HTTP_CONTENT_TYPE 2003
• ERROR_CODE_IO_BAD_HTTP_STATUS 2004
• ERROR_CODE_IO_FILE_NOT_FOUND 2005
• ERROR_CODE_IO_READ_POSITION_OUT_OF_RANGE 2008

Pooling Players

Pinterest builds the cache to pool players as needed. Historically it instantiated new player instances on the fly but that caused significant memory and bandwidth overhead. Their high-level learnings are below.

Separate by Encoding

It takes an incredible amount of work to switch contexts between different decoders. Player instances hold on to an underlying decoder reference based on the last rendered media’s encoding type. Pinterest moved toward basing player pooling on the initial decoding format. This removed latency caused by encoder switching enduring players were recycled with encoders matching the media’s encoding.

Smart Sizing

It was also a challenge to find the ideal pool size — it took multiple iterations to find the ideal space needed to house multiple video plays and avoided OutOfMemory (OOM) and Application Not Responding (ANR) errors. They employed the following two APIs to aid in this.

This callback informed Pinterest when it was time to clear out their player cache pool when they got close to sending OOMs.

setForegroundMode(true)

This flag let ExoPlayer retain a direct reference to the video decoder and keep it in memory, even in the idle state. This does take a significant toll on memory and device stability. To conserve both, Pinterest built conservative logic around the method based on the current application lifecycle and available memory on the device.

The post Pinterest: Turbocharge Android Video with These Simple Steps appeared first on The New Stack.

Sony has been developing more than video game consoles lately. In fact, its semiconductor division has been working on products designed to replace and/or reconfigure conventional IT stacks for a number of enterprises.

A prime example of this is the latest product from the Fukushima, Japan-based chipmaker called AITRIOS, an edge-based AI sensing platform that was demonstrated publicly for the first time at the National Retail Federation’s Big Show in New York, Jan. 15-17.

Between a CPU and GPU

AITRIOS is a chip with an image sensor sandwiched between a CPU and a GPU that goes inside a remote smart camera and runs on Microsoft Azure.  It takes on a full load of duties that a list of point products currently has to manage by taking images and turning them directly into usable data.

Retail use cases for AITRIOS include providing real-time data on display-shelf availability, planogram compliance, staffing, merchandising and optimizing of store operations. AITRIOS provides users with the tools, SDKs, privacy controls and development environment to collect all this data.

Another real-world example that impacts both HR policies and health regulations: It can take a video clip of an employee washing his hands in a sink, determine if he has washed for the full 20 seconds mandated by health codes, and file and store that data for future use.

AI Predictions and Modeling

“AI folks are trying to figure out how to gather physical world data into their AI predictions and modeling so retailers, industrial people working in manufacturing, petroleum industry use cases, factory automation, factory inspection, and so on can get better analytics results,” Mark Hanson, vice-president of technology and business innovation at Sony Semiconductor, told The New Stack. “A lot of products in use are old visual-inspection, rules-based systems not based on AI. They’ve been trying to figure out how to do visual AI in a democratized way where other people that don’t necessarily have a Ph.D. in data sciences can actually implement some of those things.”

At the platform layer, AITRIOS enables a combination of device integration and deployment status. It also has the ability to create models from images captured on those devices as well as deploy and manage those models, Hanson said.

“The AITRIOS platform sits underneath the application layer and the cloud layer,” Hanson said. “Typically, if you wanted to create the equivalent of this, you’d have to go find a camera manufacturer and gather data associated with the use and field of view of those cameras. Then you’d have to figure out the appropriate algorithms to run on the smart camera or build a custom camera or on the edge servers to do that. You’d have to upgrade network infrastructure to accommodate all the video that’s traveling through these.

“We know the intimacy of the camera, the image sensor, and the logic chip on it. We know how to manage them, deploy the devices through AITRIOS, and we can monitor the status and deploy models dynamically to those devices.”

All of this can be set up to “set and forget,” Hanson said, thanks to the autonomous nature of the platform.

Availability

AITRIOS is available now on the Microsoft Azure Marketplace to U.S. partners and customers, Hanson said. A free version available for testing can be found here. Lucid, a maker of industrial cameras, is the first such manufacturer to be sanctioned on the platform.

As a part of AITRIOS’s global expansion to support system integrators working on building visual AI solutions, Sony Semiconductor Services also launched its Console Enterprise Edition, a fee-based service on the AITRIOS platform, in the U.S. and Japan to meet the needs of various partners.

The post How New Sony AI Chip Turns Video into Real-Time Retail Data appeared first on The New Stack.

WebAssembly adoption is exploding. Almost every week at least one startup, SaaS vendor or established software platform provider is either beginning to offer Wasm tools or has already introduced Wasm options in its portfolio, it seems. But how can all of the different offerings compare performance-wise?

The good news is that given Wasm’s runtime simplicity, the actual performance at least for runtime can be compared directly among the different WebAssembly offerings. This direct comparison is certainly much easier to do when benchmarking distributed applications that run on or with Kubernetes, containers and microservices.

This means whether a Wasm application is running on a browser, an edge device or a server, the computing optimization that Wasm offers in each instance is end-to-end and, and its runtime environment is in a tunnel of sorts — obviously good for security — and not affected by the environments in which it runs as it runs directly on a machine level on the CPU.

Historically, Wasm has also been around for a while, before the World Wide Web Consortium (W3C) named it as a web standard in 2019, thus becoming the fourth web standard with HTML, CSS and JavaScript. But while web browser applications have represented Wasm’s central and historical use case, again, the point is that it is designed to run anywhere on a properly configured CPU.

In the case of a PaaS or SaaS service in which Wasm is used to optimize computing performance — whether it is running in a browser or not — the computing optimization that Wasm offers in runtime can be measured directly between the different options.

At least one application is increasingly being adopted that can be used to benchmark the different runtimes, compilers and JITs of the different versions of Wasm: libsodium. While anecdotal, this writer has contacted at least 10 firms that have used it or know of it.

Libsodium consists of a library for encryption, decryption, signatures, password hashing and other security-related applications. Its maintainers describe it in the documentation as a portable, cross-compilable, installable and packageable fork of NaCl, with an extended API to improve usability.

Since its introduction, the libsodium benchmark has been widely used to measure to pick the best runtimes, a cryptography engineer Frank Denis, said. Libsodium includes 70 tests, covering a large number of optimizations code generators can implement, Denis noted. None of these tests perform any kind of I/O (disk or network), so they are actually measuring the real efficiency of compilers and runtimes, in a platform-independent way. “Runtimes would rank the same on a local laptop and on a server in the cloud,” Denis said.

Indeed, libsodium is worthwhile for testing some Wasm applications, Fermyon Technologies CEO and co-founder Matt Butcher told the New Stack. “Any good benchmark tool has three desirable characteristics: It must be repeatable, fair (or unbiased toward a particular runtime), and reflective of production usage,” Butcher said. “Libsodium is an excellent candidate for benchmarking. Not only is cryptography itself a proper use case, but the algorithms used in cryptography will suss out the true compute characteristics of a runtime.”

Libsodium is also worthwhile for testing some Wasm environments because it includes benchmarking tasks with a wide range of different requirement profiles, some probing for raw CPU or memory performance, while others check for more nuanced performance profiles,” Torsten Volk, an analyst for Enterprise Management Associates (EMA), told The New Stack. “The current results show the suite’s ability to reveal significant differences in performance between the various runtimes, both for compiled languages and for interpreted ones,” Volk said. “Comparing this to the performance of apps that run directly on the operating system, without WASM in the middle, provides us with a good idea of the potential for future optimization of these runtimes.”

True Specs

In a blog post. Denis described how different Wasm apps were benchmarked in tests he completed. They included:

• Iwasm, which is part of the WAMR (“WebAssembly micro runtime”) package — pre-compiled files downloaded from their repository.
• Wasm2c, included in the Zig source code for bootstrapping the compiler.
• Wasmer 3.0, installed using the command shown on their website. The three backends have been individually tested.
• Wasmtime 4.0, compiled from source.
• Node 18.7.0 installed via the Ubuntu package.
• Bun 0.3.0, installed via the command show on their website.
• Wazero from git rev 796fca4689be6, compiled from source.

Which one came out on top in the runtime tests? Iwasm, which is part of WebAssembly Micro Runtime (WAMR), according to Denis’ results. The iwasm VM core is used to run WASM applications. It supports interpreter mode, ahead-of-time compilation (AOT) mode and just-in-time compilation (JIT) modes, LLVM JIT and Fast JIT, according to the project’s documentation.

This does not mean that iwasm wins accolades for simplicity of use. “Compared to other options, [iwasm] is intimidating,” Denis wrote. “It feels like a kitchen sink, including disparate components.” These include IDE integration, an application framework library’s remote management and an SDK “that makes it appear as a complicated solution to simple problems. The documentation is also a little bit messy and overwhelming,” Denis writes.

Runtime Isn’t Everything

Other benchmarks exist to gauge the differences in performance among different Wasm alternatives. Test alternatives that Denis communicated include:

• sightglass: for Wasmtime and Cranelift.
• PSPDFKit: (that targets WebAssembly in web browsers).
• Wasmedge’s benchmark suite.

However, benchmarking runtime performance is not an essential metric for all WebAssembly applications. Other test alternatives exist to test different Wasm runtimes that focus on very specific tasks, such as calculating the Fibonacci sequence, sorting data arrays or summing up integers, Volk noted. There are other more comprehensive benchmarks consisting of the analysis of entire use cases, such as video processing, editing of PDF, or even deep learning-based object recognition, Volk said.

“Wasm comes with the potential of delivering near-instant startup and scalability and can therefore be used for the cost-effective provisioning and scaling of network bandwidth and functional capabilities,” Volk said. “Evaluating this rapid startup capability based on specific use case requirements can show the direct impact of a Wasm runtime on the end-user experience.”

Some Wasm applications are used in networking to improve latency. Runtime performance is important, of course, but it is the latency performance that counts in this case, Sehyo Chang, chief technology officer at InfinyOn, said. This is because, Chang said, latency plays a crucial role in determining the overall user experience in any application. “A slow response time can greatly impact user engagement and lead to dissatisfaction, potentially resulting in lost sales opportunities,” Chang said.

During a recent KubeCon + CloudNativeCon conference, Chang gave a talk about using Wasm to replace Kafka for lower latency data streaming. Streaming technology based on Java, like Kafka, can experience high latency due to Garbage collection and JVM, Chang said. However, using WebAssembly (WASM) technology allows for stream processing without these penalties, resulting in a significant reduction of latency while also providing more flexibility and security, Chang said.

The post Performance Measured: How Good Is Your WebAssembly? appeared first on The New Stack.