Leading the Swedish savings market by safely liberating Kafka data
How did investment platform Avanza win > 1 million customers by enabling engineers with a secure data platform?
20 teams
on Kafka
including developers, data analysts & product owners.
Intelligent
insights
on available data and associated schema.
Good
governance
to safeguard customer data across teams & regulations.
Avanza is a leading investment and savings digital platform from Sweden, and they have data to protect. So they made it their mission to provide a secure data platform to enable developers and data engineers to collaborate more effectively.
“We want to liberate our data by design in a secure way that is sustainable for the future of Avanza. Our online systems serve our customers, so these need to be protected and kept separate from analytics. Lenses.io offers us the observability to realize this mission - de-coupling real-time operational environments from the data analytics environment by offering observability into Kafka clusters and keeping our event data flowing.”
Donny Åström Fransson, Data Engineer
Life before Kafka
Avanza's original ambition was to decouple their online services from their analytics services, putting circuit breakers in place to protect their performance and availability. After all, customer-facing apps should be safeguarded.
The company was driven by customer insights, but the underlying data pipelines needed improvement and modernization. This meant a transition from batch data processing and CSVs over FTP between different teams - to real-time data and rich integration of applications.
Apache Kafka was an obvious choice for this transition: Avanza offers a renowned customer experience to the Swedish market courtesy of their engineering practices - their engineers have an Open-Source philosophy, and the team are also fans of the event streaming architecture.
However, when the team deployed the first Kafka cluster, they spotted a big inefficiency. As they operated Kafka with Open-Source tooling, a data analyst or developer had to go to the platform team every time they wanted something.
“If open-source tooling is limiting your workflow, you’re not being strategic! The right tool for the right job sometimes means that well-placed procurement is the right option.”
Donny Åström Fransson, Data Engineer
This was a problem. Increasingly developers and analysts wanted self-service access to data and apps: they already had an effective API catalog but they needed a data catalog.
The team realized that not only were they decoupling their applications, they were decoupling their teams and products.
They needed a new DataOps approach, that puts data at the fingertips of developers, data analysts and product owners.
Initially, the Platform team considered building their own wrapper around the core of Kafka. Lack of tooling was preventing them from helping their internal customers in fulfilling their strategic initiatives. If there was no way of independently retaining an overview on what was happening in Kafka, their plan to make data-driven teams more autonomous would fail - and the platform engineers would forever be a dependency for future innovation.
Yet Kafka was powerful and complicated in equal measure, and this made self-development of tools too big of a project.
“Lenses.io has saved us a lot of time. Because Kafka is an Open-Source stack, there is a lot of tooling that you need to put together to get a complete platform.”
Anders Eriksson, Data Engineer
Apache Kafka observability once and for all
Anders and Donny wanted to build a system where developers could publish an event view of their domain once - and they could do so without direct involvement from the platform team.
They decided to use Lenses for Kafka as a window into their cluster and into the data itself.
Instead of overnight batch processing, Lenses gave a view into the real-time applications and data instantly.
At last, developers could see what they had built. With Lenses to apply DataOps practices, developers could explore data across all their Kafka streams without stress.
“Lenses gives us a wider ability to oversee Kafka. This not only helps developers to understand which data is there and how the data is represented, but provides a feedback mechanism on the schema itself. It’s a big part of our development process.”
Anders Eriksson, Data Engineer
Preventing Kafka security incidents across 20 teams
Regulations such as the GDPR have been a core data platform consideration for Avanza from the very beginning.
It is their policy to give maximum data access to different teams across the organization whilst masking sensitive information with Lenses Data Policies. Data masking is a key element of the granular permissions they have given to their developers, and each team has its own domain containing relevant data that only they should see.
The team secures Kafka with TLS and integrated secret management. Building all the other security features themselves would be too resource-intensive, so they turned to Lenses.io for support with these.
Streaming data platform for financial services
Safeguarding for security & governance with data masking, troubleshooting, data cataloguing, auditing and authentication.
App topology for good governance to see connected flows and apps in an interactive map, table or topological format.
Building applications with SQL over existing infrastructure - for example Kafka or Kubernetes cluster.
Multi-dimensional troubleshooting using insights into app performance & flows to explore topics, replaying data or skipping events.
“Lenses gives greater observability over our Kafka cluster, saving a lot of time for our Back-end Developers and Data Engineers in their quest to get a quick overview of the available data and associated schema. This is key to creating a truly innovative and fun environment working with data.”
Fredrik Thulin, Product Owner
Democratizing data means results for everyone
The ability for anyone to fast-forward and replay through data in time, as well as seeing it in a compact format like a table or topology, was a dealmaker for Anders and Donny when it came to working with Lenses.io.
With 20 teams already working with Kafka, Avanza's engineering team is now on a path towards total self-service and data socialization across the company.