RedisConf19 - Instant - DataOps on Redis

I’ve just returned from the Redis conference in San Francisco, it was my first time at a Redis Conference and I must say I’m very impressed by the whole Redis stack. I also had the honour of participating in the keynote to show off Lenses SQL on Redis, a big thank you to RedisLabs for making this happen. I also presented a session on DataOps and Lenses SQL. You can find the slides here.

Here’s a few notes I took.

What did I learn, from a CTO perspective?

Redis can now be used to build the underlying infrastructure for a data platform, not only is Redis Core available with its legendary low latency and scalability but the new multi-model proposition offers graph, document, search and key value. This provides a solution for the persistence layer that plagues many platforms .i.e. how to provide persistence and query capability for different access patterns without managing and maintaining multiple different stacks.

Multi-model is made up of the different modules Redis provides and the glue is a new module, Gears. Gears is a serverless engine for performing data processing, supporting multi-model and Streams that was introduced in 2018. Gears works like Lenses SQL runners, that Lenses can deploy to Apache Kafka Connect and Kubernetes.

So now you have Redis core, multi-model, a transport middleware layer in Streams and Gears to processes real-time data. They also have Redis Edge to drive IoT applications. Wow.

So where does Lenses help fit in?

In a data-driven world, data is the protagonist and empowering data users is a primary goal. Providing simplified access without the requirement for specialized and hard to find developers is a must.

According to Stripe;

68 % of organisations believe access to developer talent is a threat to the success of their business

To be data-driven, every data literate person in your organizations needs access to your data, governance permitting, not just developers. This means visibility on the underlying systems, like Redis. Lenses provides this access.

We recognize there’s more than just Apache Kafka for streaming, Redis shows this with Streams but they also have a wider offering which is compelling. With Lenses SQL now working over Redis, we are now starting our journey, as an overlay provider, to deliver DataOps capabilities over all streaming data regardless of the underlying infrastructure. Interestingly, Redis Stream API bindings are on the way for Java/Scala which happens to be the languages the Lenses SQL runners are written in.

DataOps isn’t only about visibility, there’s operations, monitoring and also Data Ethics, governance and auditing. Naturally, with Lenses SQL on Redis this is applied out of the box with Lenses Data Policies.

Want to migrate your data out of Kafka to Redis?

Then we also have SQL in our Redis Sink Connector, which is part of our open source Stream Reactor project Download the latest Lenses release and join our community slack and we can chat further.

Download Lenses now to give is a spin!