Deploy Lenses on AWS EKS for MSK
In this blog we will describe how to deploy Lenses on EKS on top of AWS Managed Streaming for Apache Kafka, a fully managed and highly available Apache Kafka service.
The Cloudformation template that deploys Lenses in EKS will require that you have configured an EKS cluster, an MKS cluster and the appropriate roles that are required by Lambdas to describe MSK and manage EKS.
This guide expects that the following items have already been configured:
Create a Lambda Role for accessing EKS and MSK
We need a role which will allow AWS lambdas to:
Describe MSK Cluster (Get broker endpoints, cluster’s arn, etc.)
Authorize with EKS cluster for Managing Deployments
Go to AIM, then roles and select
Add the following policies:
AWS managed policy
After creating the role, you need to map it explicitly into the EKS cluster auth configmap since only the creator of EKS cluster has access by default to the cluster. Any other user/role must be added in advance.
First, get the
rolearn and then type in your terminal:
kubectl edit -n kube-system configmap/aws-auth
Append the following under the mapRoles key:
- rolearn: <RoleArn>
<RoleArn> with the
rolearn id of the role you just created above and then save and exit. Allow EKS and Worker Nodes to communicate with the brokers
While both EKS and Worker Nodes are sharing a common network, it is not enough for the clients deployed in EKS to communicate with the brokers. This is a security feature designed by AWS that requires adding the security group of a specific resource in the inbound tab of MSK’s Security Group in order for the resource to be able to communicate with the brokers.
Visit the MSK Cluster and select the security group under Networking section/Security Groups. Click on the inbound tab and select Edit, then select Add Rule, choose All Traffic as type, paste the security group id of the EKS Workers NodeGroup and click save.
That is all – you are done! Now EKS worker nodes and their resources should be able to communicate with the Kafka brokers.
The final phase is to deploy Lenses, which should also be straightforward. Firstly, request a free license key.
After that, go to Cloudformation, import the EKS lenses.yaml template and click next. Give a name for the stack, and then provide the following parameters:
EKS Cluster Name
MSK Cluster Name
S3 Bucket Name *1
S3 Bucket Prefix
Lambda Role *2
The S3 bucket name and S3 bucket prefix are the bucket and its prefix that will be used to copy the Lambdas from our public S3 bucket to your bucket.
The Lambda Role is the role you created and added into the EKS ConfigMap (See Create a Lambda Role for accessing EKS and MSK).
When you provide all the required parameters, click Create stack. Deploying the stack should take around 5 minutes to complete. When the deployment is finished, go to the output tab, copy the value of
LambdaEKSConfigLensesEndpoint, open a new tab, paste the content and hit enter.
You should now be welcomed with the Lenses Login screen.