TriggerMesh: Open Sourcing Event-Driven Applications
Event-driven architectures are reshaping the way information is shared between the services producing it and those consuming it. Systems idling and waiting for data are becoming more and more obsolete, as software architectures continue to transition from API-driven to event-driven, and as real-time event streaming continues to grow in popularity.
As companies, projects and infrastructure grow, so does the need for more multicloud, real-time data flows. Enterprises demand simpler workflows for DevOps engineers, real-time alerts when testing is complete, and tracking security logs between different Software as a Service (SaaS) applications.
All are necessary in today’s growing cloud native ecosystem, and without open source tools or the confinement of vendor lock-in, they’re tough to find.
TriggerMesh is an serverless event router with one simple main goal: to process events in real time and route them to the correct consumers. It’s a multi-purpose tool with an ever-growing list of use cases.
TriggerMesh’s open source integration platform provides users with a configuration-driven solution that will feel familiar to users of Infrastructure as Code (IaC) offerings like Ansible, Chef and Terraform. Built on top of Kubernetes, it includes a declarative API to quickly and programmatically connect data and services from multiple sources into event-driven applications.
Sources for message consumption include Amazon Web Services (AWS) SQS, Google Pub/Sub, Azure Event Hubs, and Kafka. TriggerMesh filters and transforms messages, combines processing capabilities with serverless functions, and connects them with messaging sinks linking Elasticsearch, AWS Simple Storage Service (S3), and Apache Kafka.
TriggerMesh connects to legacy service buses as well to create workflows between applications or data sinks to other systems such as Azure Data Lakes or Snowflake.
“Often people will use it to do things like trigger a function, or service, or … microservices,” Jonathan Michaux, product manager at TriggerMesh, told The New Stack.
TriggerMesh can also play a role in a larger streaming ecosystem, he added, such as providing “a function [for] processing events and pushing them to another system like Kafka for further processing to happen later on.”
For a quick comparison, think of Amazon’s EventBridge. Introduced in 2019, the event router allows developers to write rules that take action once an event occurs in Amazon’s S3 bucket.
TriggerMesh is the open source alternative, offering similar capability without vendor lock-in requirements. More recently, TriggerMesh introduced its latest open source offering, Shaker, which continues the push toward more widespread adoption of TriggerMesh. Shaker removes the Kubernetes requirement and provides the capability for TriggerMesh to run on Docker.
Open source doesn’t just offer freedom from vendor lock-in. It means TriggerMesh is completely cloud agnostic, allowing users to produce and consume between multiple clouds and on-prem data centers.
“Think about what this is and how powerful it could be if we could just take any application and connect it through events in such an easy way,” Michaux said. “You can deploy it anywhere you want on any Kubernetes cluster or on any machine that has Docker on OpenShift, on Amazon, on Azure, or Google.”
Michael Edenzon, co-founder of Fianu Labs, a software governance company, told The New Stack that TriggerMesh’s cloud agnosticism “absolutely moves the ball forward when it comes to making it easier for developers to build event-driven applications.”
Case Studies and Use Cases
How are real organizations using TriggerMesh in their event-driven architectures? Here are two examples.
ManoMano: Saving Costs on Running Microservices
ManoMano, a home-improvement marketplace based in Europe, is an e-commerce site that offers web and mobile experiences. Its many business needs require hundreds of microservices. With a codebase that large, it relies on diverse performance testing.
By adding TriggerMesh to its stack, ManoMano’s site reliability engineering (SRE) team now had the correct software tool to provide a serverless eventing experience that allows test developers to execute code when specific AWS events occur, by having the test developers subscribe to specific events.
More recently, ManoMano dove into its long-running microservices. Do they need to be long-running and always up? Does their usage justify the continuous consumption of compute resources and associated costs?
The answer was no, which drove the company’s desire to start running services on an as-needed basis only. The investigation revealed that many of these services only process a few requests per day, and most of their time is spent idling.
ManoMano replaced long-running idle tasks with containers scheduled on-demand to run EKS in reaction to AWS S3 events. This was made possible thanks to the new event-driven architecture, which gathers events from AWS services, ingests them into a centralized broker, and allows developers to subscribe to specific events for consumption. The platform abstracts security and infrastructure concerns away.
Fianu Labs: On-Prem Automated Software Governance
Fianu Labs’s fully automated, event-driven software governance tool provides instrumentation throughout the CI/CD process and captures events throughout the software development lifecycle and compares them against predefined policies and automated compliance documentation.
TriggerMesh, which sits on top of Google Kubernetes Engine with Knative, is a key part of Fianu’s architecture and is woven through the fabric of its workflow. Though not operating at huge scale, Fianu requires redundancy and reliability while supporting 200,000 code repositories, which roughly breaks down to 500 to 1,000 events per minute.
“The TriggerMesh tooling and the way that we’re using it allows us to keep a robust, event-driven system and keep it pretty neatly,” Edenzon, of Fianu Labs, told The New Stack.
An example of the typical TriggerMesh workflow inside Fianu starts with the ingestion of events. As events are ingested, Fianu uses TriggerMesh functions (Fianu uses embedded Python but TriggerMesh has Node.js and Ruby options as well) to make transformations and computations on the payloads.
Edenzon discussed the example of changing strings to floats to ensure the incoming data is consistent with policy. While there may be one incoming source of data, a number of different outputs could be needed and TriggerMesh functions abstract away the need to create core tooling.
“Without creating a serverless function, we can embed the TriggerMesh functions to perform small bits of business logic along the way,” he said.
TriggerMesh Targets was another aspect of the TriggerMesh functionality that Endezon pointed out. Targets allow Fianu’s customers to build customizations on top of the Fianu software as it aids with complicated event destinations, by abstracting away the complexity for events going to a Kafka stream or to slack.
TriggerMesh doesn’t only help Fianu developers, it also helps the company’s customers. Fianu provides a lot of out-of-the-box capabilities to its end users, and while the company isn’t offering specific TriggerMesh tooling but because of TriggerMesh’s high level of abstraction, customers can write custom plugins as Knative functions in any language and snap them into the running instance, because TriggerMesh treats them as just another event processor.
“As developers, if we were only ever required to just focus on the business functionality, we’d be extremely productive,” Edenzon said. “So that’s one of the things that we like about [TriggerMesh]. It makes it very easy for us to isolate and test business functionality.”
Embedded into SaaS Apps
TriggerMesh makes it possible for SaaS applications to ingest events from multiple cloud providers, transform the events to match specific schemas, and then send those events to any destination.
“TriggerMesh makes it very easy to ingest new event sources into an application,” Michaux said. He added that “B2B SaaS vendors can extend their app’s ecosystem, while keeping development teams focused on the core product.”
In cybersecurity, this means the transformation of different cloud security events into standardized schemas, such as the recently announced Open Cybersecurity Schema Framework, before sending them to the customer’s preferred security solution for analytics and threat detection.
The value for the security community is incredibly high. Since security threats happen in real time, the value isn’t only in having a collection of data or integrating applications, it’s about processing events as they occur.
“A typical example would be someone making a failed login attempt into a system, for example a cloud provider like Oracle or AWS,” Michaux said. “That’s an event that a threat detection system would be interested in analyzing. TriggerMesh can help ingest the event and deliver it to the security system, in real time, in the right format, with minimal fuss.”
Conclusion
TriggerMesh is an example of a use case meeting demand and allowing demand to speed up to interact with the technology. It’s open source, with the only requirement now being Docker with the newly released Shaker making adoption even more available.
“When using TriggerMesh to its fullest, with the right application design, it allows you to isolate the business functionality without having to worry about the mechanics of how data goes where,” Edenzon said. “So you can just focus on the business functionality”.
