Navigating Streams with Kubernetes
Speaker(s): Christopher Kalika
Redis Streams provide a scalable, flexible, and efficient way for a group of clients to access a flow of information. We decided to create custom resource definitions (CRDs) for streams, queues, and related functionality in Kubernetes. This undertaking is part of a process of replacing and extending the native job processing functionality, within Kubernetes, with a Redis-based system. We are big believers in optimising our use of resources, so we’re leveraging Redis’ pub/sub functionality to instantiate new containers, as and when they are needed. (i.e. a single process can chew threw as many jobs as are available, but is only instantiated as and when). This aligns well with modern serverless approaches, and allows us to scale as we need to.
We’re looking at writing some Redis modules to handle 1. Transparent encryption of our data, 2. Segregation of data, and 3. Transactional handling (i.e. encrypt it, store it, hash it, log it to STDOUT, notify a PubSub channel, and put it in one or more streams). The goal of this to provide multiple tenants with a cluster, across a variety of tech stacks, with a consistent approach to this. We’re still not sure if it’s the right approach, but we’re going to give it a try, anyway.