Redpanda Lowers the Complexity and Cost of Streaming and Event Data

I recently wrote about the need for enterprises to harness events to process and act upon data at the speed of business. The core technologies that enable enterprises to process and analyze data in real time have been in existence for many years and are widely adopted. However, streaming and events technologies are also commonly seen as a niche requirement, separate from an enterprise’s primary focus on batch processing of data at rest. One of the reasons for this is an entrenched reliance on batch data processing products and workflows. Another is the high-level expertise that has been required to implement and maintain streaming and event processing technologies. In recent years, many software providers, including Redpanda, have set out to lower the barriers to working with streaming and event data to encourage wider adoption.

Redpanda was founded in 2019 by CEO Alexander Gallego with the goal of making real-time data accessible to all developers, rather than just those with high levels of expertise related to streaming and event data processing systems and architecture. Based on their experience with existing streaming and event data processing products, Gallego and his team knew that the cost and complexity of implementing streaming and event technologies was high, presenting a barrier to adoption. They also knew that a higher number of use cases and greater data volumes resulted in increased cost and complexity, which put the brakes on real-time data processing playing more than niche role in many enterprise data strategies. ISG’s Analytics and Data Benchmark Research highlights the relatively low level of adoption of streaming data, with less than one-third (30%) of participants using streaming data in analytics. Redpanda’s goal was to create a streaming data platform that would be compatible with the Apache Kafka distributed messaging and event streaming platform without the operational complexity that Kafka was then known for. The product was initially introduced in 2021 at the same time as Redpanda’s announcement of $15.5 million in funding from Lightspeed Venture Partners and GV. Redpanda subsequently announced a further $50 million of Series B funding in 2022, followed by a $100 million Series C round in 2023. In addition to the development of the company’s streaming data platform and managed cloud services, Redpanda has also invested some of its funding in acquisitions, including the web user interface expertise of CloudHut in 2022 and streaming data integration connector developer Benthos in May 2024.

I assert that by 2026, two-thirds of enterprises will require streaming and event data processes with low latency of seconds or sub-seconds to satisfy operational requirements. A key first step Gallego took to diminish the cost and complexity of streaming data technology was to reduce the resources required to deliver high-performance data processing. To achieve the desired performance gains, the Redpanda streaming data platform was written in C++, with each node in a cluster designed to be a self-sufficient single binary that natively implements the Raft consensus protocol for data management and control across the cluster. These design choices resulted in a reduced footprint compared to existing streaming and event systems and avoided dependency on Java virtual machines and external distributed coordination systems such as Apache ZooKeeper. Additionally, Redpanda implemented a thread-per-core programming model designed to take full advantage of available hardware resources (including processor, memory and disk) and avoid over-provisioning hardware to meet scalability requirements. Redpanda was also designed to take advantage of cloud object storage by default, with support for tiered storage combining cloud and local storage resources. The product also includes the Redpanda Console user interface, which is built on the capabilities acquired with CloudHut and the delivery of more than 280 pre-built integration connectors via Redpanda Connect (built on the Benthos acquisition). Redpanda is delivered as self-hosted software, as well as via a range of cloud managed services.

Redpanda Cloud is available as dedicated clusters hosted and managed by Redpanda on single-tenant AWS, Google Cloud or Microsoft Azure cloud infrastructure, as well as serverless clusters hosted and managed by Redpanda on multi-tenant shared cloud infrastructure on AWS. Redpanda also supports bring-your-own cloud (BYOC) clusters, in which the data plane is hosted on the customer’s cloud infrastructure provider but provisioned, monitored and maintained by Redpanda via its control plane in Redpanda Cloud. The BYOC offering is designed to support low-latency data processing needs as well as compliance with data sovereignty, security and privacy regulations. While many cloud providers have policies that pertain to protecting data privacy, emerging data sovereignty requirements place additional burdens on enterprises to not only have reassurances about how the data is stored and processed, but control over the infrastructure used to store and process the data. Separating the control and data planes also enables an enterprise to assert more control over the performance of the data processing software and enforce multiple layers of security without ceding permissions to the software provider. BYOC can also enable enterprises to avoid cloud-egress costs that would otherwise be involved in moving data to the software provider’s environment.

Sovereignty is also a concept that is associated with Redpanda’s nascent efforts to address requirements for artificial intelligence (AI), with the company looking to combine AI models with data streaming workloads in virtual private cloud environments to enable private inferencing, as well as AI lineage tracing and integration with role-based access control. Redpanda’s Sovereign AI approach is in early preview, and we anticipate further details and enhancements being rolled out during 2025. We also look forward to more details about Redpanda One, the company’s forthcoming multimodal streaming data engine, which is being designed to enable users to define data storage for each individual topic based on its unique requirements for availability, consistency, latency, safety and cost-effectiveness. In the interim, I recommend that any enterprise evaluating options for streaming data and event processing include Redpanda in its evaluation.

Regards,

Matt Aslett