“Where do we need to take our data platform to enable our developers?"
We want to liberate our data by design in a secure way that is sustainable for the future of Avanza. Our online systems serve our customers, so these need to be protected and kept separate from analytics. Lenses.io offers us the observability to realize this mission - de-coupling real-time operational environments from the data analytics environment by offering observability into Kafka clusters and keeping our event data flowing.”
Donny Åström Fransson, Data Engineer
Life before Kafka
Avanza's original ambition was to decouple their online services from their analytics services, putting circuit breakers in place to protect their performance and availability. After all, customer-facing apps should be safeguarded.
The company was already driven by customer insights, but the underlying data pipelines needed improvement and modernization. This meant a transition from batch data processing and CSVs over FTP between different teams - to real-time data and rich applications integration.
Apache Kafka was an obvious choice for this transition: Avanza offers a renowned customer experience to the Swedish market courtesy of their engineering practices - their engineers have an Open-Source philosophy, and the team are also fans of the event streaming architecture.
Deploying the first Kafka cluster awakened a greater need for the team, however. They initially started operating Kafka with Open-Source tooling. Every time a data analyst or developer wanted something, they had to go to the platform team.
“If tooling is limiting your workflow, you’re not being strategic! The right tool for the right job sometimes means that well-placed procurement is the right option.”
But increasingly developers and analysts wanted self-service access to data and apps: they already had an effective API catalog but they needed a data catalog.
The team realized that not only were they decoupling their applications, they were decoupling their teams and products.
They needed a new DataOps approach, that brings best practises and puts data at the fingertips of developers, data analysts and product owners.
Initially, the Platform team considered building their own wrapper around Kafka. But they learned that there is a lot of functionality to develop around the core of Kafka, so it became too big a project. They realized that this was preventing them from helping their internal customers in fulfilling their strategic initiatives.
Yet Kafka was powerful and complicated in equal measure.
In short, if there was no way of independently retaining an overview on what was happening in Kafka, their plan to make data-driven teams more autonomous would fail - and the platform engineers would forever be a dependency for future innovation.
“Lenses.io has saved us a lot of time. Because Kafka is an Open-Source stack, you discover that there is a lot of tooling that you need to put together to get a complete platform. For example as soon as you start consuming data you realize you need to monitor consumer groups and then you look for another tool.”
Anders, Data Engineer
Apache Kafka observability once and for all
Anders and Donny wanted to build a system where developers could publish an event view of their domain once - and they could do so without direct involvement from the platform team.
They decided to use Lenses.io as a window into their Kafka cluster and into the data itself. Instead of overnight batch processing, the Lenses workspace on Apache Kafka gave a view into the real-time applications and data instantly.
At last, developers could see what they had built. With Lenses DataOps workspaces, teams benefited from the stress-free experience of having a consistent way for Developers to explore data across all their Kafka streams.
Data was within touching distance; and luckily it looked healthy.
“Lenses.io has given us a wider ability to oversee Kafka, helping our developers to understand which data is there. It’s a big part of our development process. Our teams use Lenses.io as a portal to provide insights into how the data is represented, but also as a mechanism for feedback on the schema itself.”
Anders Eriksson, Data Engineer
Preventing Kafka security incidents across 20 teams
Since the beginning of their Kafka project, regulations such as the GDPR have not been an afterthought for Avanza - but a core data platform consideration from the ground up.
To meet this end, it is their policy to give maximum data access to different teams across the organization whilst masking sensitive information with Lenses Data Policies. Data masking is a key element of the granular permissions they have given to their developers, and each team has its own domain containing relevant data that only they should see.
The team did secure Kafka with TLS and integrated secret management. Building all the other security features themselves would be too resource-intensive, so they turned to Lenses.io for support with these.
“That view when you look into a topic, when you see it in a compact format like a table or a topology - and the fact that you can fast-fwd back and forth through data in time. That was a dealmaker for choosing Lenses.io.”
Donny, Data Engineer
Democratizing data yields results for everyone
With 20 teams already working with Kafka, Avanza's engineering team is now on a path towards total self-service and data socialization across the company.
“Leveraging Lenses.io to give greater observability over our Kafka cluster saves a lot of time for our Back-end Developers and Data Engineers in their quest to get a quick overview of the available data and associated schema. From a business point of view I believe this is key to creating a truly innovative and fun environment working with data."
Fredrik Thulin, Product Owner