Fruit-Picking Robots Powered by Kubernetes on the Edge
How do you deploy Kubernetes on the edge in areas of remote connectivity and little to no bandwidth? With low-to-no supervision? How about deploying, updating and securing a fleet of AI-enabled flying robots to pick fruit?
These are the challenges Tevel’s next-gen harvesting solution is up against as it looks to limit food waste and feed more people with less people. As about 30% of the world’s fruit is not picked in time, due to logistical challenges and seasonal labor shortage, Tevel looks to fill that gap with flying autonomous robots already deployed across Italy, California and Washington state.
The smart agriculture or AgTech industry is growing year over year, as a way to increase food production and security alongside the scaling global population.
DevOps leader Itzik Ben Zaken spoke of Tevel’s software plus hardware plus the edge journey during the Kubernetes on the Edge day as part of KubeCon+CloudNativeCon Europe. He was joined by Pedro Oliveira, solutions architect at Spectro Cloud. Together their teams collaborated over a solution.
We break down the sociotechnical challenges and the technical solutions — including Kubernetes on the edge — now.
The Challenge of Launching a Swarm of Robot Fruit Pickers
First, the setup. Tevel’s autonomous robots are connected via power supply and networking cable to central units that process the sensors and sophisticated AI that searches for, navigates to, and detects the fruits, very gently picking them when ready. The flying robots also act as real-time data agents, gathering information about fruit weight, color grading, ripeness and disease detection. Also, the two-thirds of the year the fruit isn’t locally in season, the multifunctional robots can be deployed to do other things like pruning and precision spraying.
Also looking to help address the global bee shortage, Zaken later told The New Stack that Tevel is working with bio-mimicking pollination tech company BloomX on autonomous pollination of avocado trees.
This is all achieved autonomously. Unsurprisingly, there were many technological challenges to overcome to make the solution happen.
“As much as we want our robots to be autonomous, we want our applications to be autonomous,” Zaken said. “You need something like Kubernetes to do this job because fruits are not waiting for software updates.”
But these Kubernetes clusters are in fields, far away from 5G and antenna towers. These swarms have to work in spite of unstable connectivity, low or no supervision, access difficulties, risk of theft, and being very difficult to deploy.
Tevel’s engineering team is based in Tel Nof, Israel, so they can’t be flying halfway around the world to set anything up or to fix something. That would just be another day of fruit picking wasted. They have to be able to manage the Kubernetes clusters in the field, with no network and connectivity instability. And be able to do it all via remote installation.
“As a startup, you need to run, and run fast, so each developer, at the beginning, had their own robot,” Zaken said. At first, they each needed to connect an SSH or Secure Shell network communication protocol and manually scrape to remote control each robot, with an x-server to get video, using bash scripts to control the robots remotely.
What they ended up with was different setups for each engineer and each host. The robot setup was manual with the operating system and applications, and it was painful, Zaken said, giving the whole team a bad case of “works on my machine” syndrome.
This created a high onboarding cost for both the devs and ops teams, and “it was impossible to operate multiple drones. It was too many screens,” he continued.
Initial Challenges of Kubernetes on the Edge
About four years ago, the Tevel team decided to embrace containers. They implemented Docker for easier setup and application, and to “freeze our setup.” The Tevel team built a small web user interface to control everything. But they still had a lot of challenges.
“When you’re handling containers manually, it’s difficult. Our images were huge, heavy, and the build was difficult,” Zaken said. At the time, deployment was separate between the deployment station and each robot. Even with containers, the robots were still hard to operate.
So they decided to try Kubernetes. Tevel divided services into different containers. Finally, the deployment lifecycle streamlined, the robots (and their humans) were able to embrace DevOps by combining deployments with the operation station. Immediately, with microservices architecture, the images became slim and thin. They were also able to add service monitoring and controls to the web UI.
“Multi-robot operation was now a breeze,” Zaken said.
But they initially found a lot more challenges with Kubernetes on the edge, at their scale of millions of metric tons. It was complex to manage multiple host installations at scale, he said, and gathering statuses for all assets was too manual. They were still unable to be sure the cluster was running all the time, when far away from antennas, he continued, and operating multiple platforms was just plain hard.
Since security work is never finished, extra security needs were required like operating system compliance and manual encryption — which Zaken noted is particularly painful at the edge.
Most importantly, they wanted to reduce the cognitive load on their team, and to streamline a more out-of-the-box developer experience.
“At Tevel, we wanted to focus on the application, on picking fruits, not the update system,” he explained.
Tevel Partners with Spectro Cloud on the Edge
Tevel began working with Spectro Cloud to find these next solutions. The partnership zeroed in on four needs shared by the hundreds of these geographically distributed, multi-purpose robots:
- Easy provisioning of operations
- Functional when disconnected
- Zero risk remote upgrades
- Full-stack security
This stack includes the ground control station — a Kubernetes cluster that is run by an operator and can control any number of bins. That station is wirelessly connected to bins that can then manage up to eight multipurpose robots for fruit picking or pruning.
In order to manage Kubernetes clusters at scale at the edge, they adopted Spectro Cloud Palette Kubernetes lifecycle management platform, which works across data centers, public clouds, the edge and a combination of these options.
On the edge, Oliveira from Spectro Cloud explained at KubeCon, Palette manages the highly distributed architecture, with each cluster running its own Palette agent, separating the management plane from the control plane. This was designed specifically for such disconnected use cases with no bandwidth.
“Like Itzik was saying, they don’t want to be playing around with OS dependencies, having to install Nvidia dependencies, kernel dependencies, all the way to the application. And that includes also all the integration in between,” Oliveira explained. “With Palette, you specify the desired state of your cluster, which we call a cluster profile.”
For about 90% of edge use cases, Oliveira said, connectivity and security are always an issue, which makes provisioning — setting up and deploying a device — a persistent challenge.
Within Palette, you specify a cluster profile. Then you apply that cluster profile to a cluster. And then the cluster declaratively reconciles and manages itself with the desired configuration and policies. Spectro Cloud calls this easy provisioning, at the edge.
Tevel may have very short notice that fruit is ready for the picking, so they needed the ability to get robots shipped to and provisioned at distribution centers that aren’t set up with the right equipment or staffed with the right technical skills.
To this end, Spectro Cloud’s Palette uses Kairos, the open source, Linux meta-distribution for edge, to build an immutable operating system and Kubernetes images — in Tevel’s case, based on Ubuntu and K3s. K3s is a Kubernetes distribution designed for the edge and the Internet of Things — unattended, resource-constrained production workloads in remote locations that require high availability.
“Once you boot up and install your device, you’ve only got to ship it out and the operator has to power it on and it connects. From then on, he can register the device with Palette,” Oliveira explained. Once that device is registered, they can build a cluster. All of Tevel’s operations are run through Spectro Cloud Palette.
Out in the fields, the robotic devices come with QR codes that trigger bin and edge device registration via Palette, including:
- Edge machine ID
- Device name
- Farm ID
- Deployment region for multi-agency
But not all the operators in these rural environments have the connectivity or even the screens to register these devices. Or to update registration if the hardware fails, requiring a quick swap out.
For these cases of connection disruption, there is an added requirement of headless, zero-touch provisioning, without a registration, with all the above information provided in advance, alongside an edge token that is authenticated against Palette. Then the device is registered automatically with Palette, even if not online.
Another challenge is when the environment is fully disconnected — on what is called a gray site, or an air-gapped deployment. When there’s no connectivity between the field and Tevel’s headquarters, how can Tevel not only maintain these robots, but even know if the functionality is still there? Because Palette is intentionally built with distributed architecture, the edge nodes are separated from the control plane and management plane.
“Once the cluster is running and everything is going, the cluster is managing itself. So even if there’s no connectivity back to the management platform, the node, the device, and the cluster, will still maintain its status, its applications, its configurations, and all of its policies as well,” Oliveira explained.
Kairos is a container-based image that splits a disk into A/B or active and inactive partitions. If an upgrade fails, it automatically reverts back to the previous configuration, so the upgrade doesn’t break the workflow. This is referred to as a zero-risk upgrade.
With Kairos, the operating system is also immutable, or unable to be changed, with read-only roots to provide extra security, so nothing can be installed on an individual cluster or device.
With these robots, the cost of burglary is a consideration beyond the device but to the intellectual property too, particularly when they are offline. It wouldn’t be scalable to have to generate YubiKeys for each of 60 bins (the groups of up to eight robots.)
The Tevel-Spectro Cloud collaboration decided to use a Trusted Platform Module or TPM to deliver persistent data encryption. Oliveira explained that the TPM generates unique one-way hash keys that are saved into the hardware. Both a TPM and a key management service (KMS) are required to boot. Every time the device boots, it challenges the TPM, and will only then turn on if that challenge is successful. This means that the disks remain encrypted if a device is stolen.
Zaken says Tevel’s mission is to help feed the world. They aim to achieve this by helping farmers be more profitable by always being able to pick every piece of fruit on time, any time of day, while also gaining real-time data on each piece picked. The Tevel team doesn’t want to get distracted by hardware and connectivity issues. They want to focus on increasing the overall efficiency and reliability of their process.
This tech setup “reduced the time that we spent on building and managing clusters by ourselves,” Zaken said. “And it frees our engineers’ time to be focused on innovation” around robot fruit picking.
Introducing SENA: Secure Edge-Native Architecture
Next step is they want to reduce their infrastructure cost, getting rid of the ground control station level.
And of course are looking to increase security throughout the stack, implementing the Secure Edge-Native Architecture (SENA) framework announced at KubeCon earlier this week. SENA is the result of a collaboration between Spectro Cloud and Intel which applies zero trust principles for deep-root security at all layers of edge computing environments. SENA integrates the best-of-edge technologies from Intel, Spectro Cloud and open source, including Intel Smart Edge, Spectro Cloud’s Palette, and Kairos.
SENA offers a holistic pathway for advanced security across the full-edge lifecycle from silicon to app:
- Deploying trusted hardware – how to know a device hasn’t been tampered with — device onboarding and authentication
- Provisioning a verified software stack to the hardware – how to know that the operating system, Kubernetes, and application are all as intended (software bill of materials or SBOM) and free of vulnerabilities (across the supply chain levels or SLSA)
- Operating runtime securely – including verifying the boot, image immutability, confidential computing, encryption in use and in transit, and workload isolation
- Managing the edge lifecycle – including monitoring, patching, upgrades, zero trust role-based access control (RBAC), and self-healing reconciliation loops
By applying these next steps for security on the edge, Tevel can focus on making sure each ripe fruit is picked and every possible side job for the robots is optimized.
Check back often for all things KubeCon+CloudNativeCon Europe 2023. The New Stack are your eyes and ears on the ground in Amsterdam!
