Deploy turn-key DataOps for AWS MSK

Running your own Kafka is starting to feel like wading through oatmeal.

We’re not the only ones thinking that.  The majority of organizations we speak to have or are in the process of moving their Kafka to a managed service.  

If you’re already an AWS-shop, Managed Streaming for Apache Kafka (MSK) is a no-brainer. It is the same Kafka that we know and love and integrated with other AWS services such as IAM, Cloudwatch, Cloudtrail, KMS, VPC and more. And of course you can run your data apps on EKS. 

A managed Kafka takes the heavy lifting away from your infrastructure operations, but that's only half the battle. At Lenses.io we've built the tools you need for data discovery, governance, self-service to take more load away from your data operations (DataOps). Including the support for AWS Open Monitoring.

A two-click AWS MSK provision

In a matter of two clicks in the AWS console, you can have your fully managed Kafka MSK cluster provisioned.

So AWS set the bar high for us to deploy Lenses DataOps for MSK just as seamlessly.

Traditionally, there have been two ways of deploying Lenses for MSK. Manual install via Docker/Helm/Tarball or via the AWS Marketplace which triggers a CloudFormation template.

In a bid to deliver a quicker and better experience, we’ve delivered a Cloud Portal to deploy instances of Lenses (“workspaces”) across different clouds. And we’re prioritising AWS and MSK as our first cloud.

The Portal deals with all the management of deploying Lenses on an EC2 instance and connecting it to your MSK Cluster, Schema Registry and Kafka Connect Cluster (should you have them). It’s also taken out the need of requesting a license key.

Here’s how you deploy Lenses DataOps for MSK in 5 steps

1. Create a portal.lenses.io account

With a single Lenses.io Cloud Portal user account, you can deploy instances of Lenses and operate them centrally. In the current release, two deployment types are supported: a demo environment with pre-configured Kafka and datagens, or a Lenses workspace for MSK. 

2. Create an IAM role

For Lenses to be able to automate the deployment of Lenses in your VPC (Amazon Virtual Private Cloud), it must have credentials to access certain of your AWS resources on your behalf. 

This is a common practice amongst AWS partners. The recommended option from AWS involves you creating an IAM role granting us (a third party) access to your AWS resources. This role will be used to generate temporary credentials to access certain resources. 

The role can be created via a Cloudformation template that has already been prepared:

The portal will redirect you to your AWS console so that this role can be created.

What does this role permit Lenses to do?

The IAM role called LensesioIntegrationRole will have access to these resources:

• iam:CreateRole

• iam:PassRole 

• iam:CreateInstanceProfile 

• iam:AddRoleToInstanceProfile 

• iam:DeleteInstanceProfile 

• iam:GetInstanceProfile 

• iam:PutRolePolicy 

• iam:DeleteRolePolicy 

• iam:RemoveRoleFromInstanceProfile 

• ec2:DescribeTags 

• ec2:CreateTags 

• ec2:DescribeInstances 

• ec2:RunInstances 

• ec2:DescribeVpcAttribute 

• ec2:DeleteTags 

• ec2:AuthorizeSecurityGroupIngress 

• ec2:AuthorizeSecurityGroupEgress 

• ec2:CreateSecurityGroup 

• ec2:DeleteSecurityGroup 

• ec2:DescribeSecurityGroups 

• ec2:RevokeSecurityGroupEgress 

• ec2:RevokeSecurityGroupIngress 

• ec2:UpdateSecurityGroupRuleDescriptionsIngress 

• ec2:DescribeVpcs ec2:DescribeSubnets 

• ec2:DeleteVolume 

• acm-pca:GetCertificate 

• acm-pca:IssueCertificate 

• cloudformation:CreateStackInstances 

• cloudformation:DeleteStackInstances 

• cloudformation:UpdateStackInstances 

• cloudformation:DescribeStackResource 

• cloudformation:DescribeStackEvents 

• cloudformation:DescribeStacks 

• cloudformation:ListStacks 

• cloudformation:CreateStack 

• cloudformation:DeleteStack 

• logs:CreateLogStream 

• logs:DeleteLogStream 

• logs:CreateLogGroup 

• logs:DeleteLogGroup 

• logs:DescribeLogGroups 

• logs:DescribeLogStreams 

• logs:PutLogEvents 

• kafka:Describe* 

• kafka:Get* 

• kafka:List*

This access is required for Lenses to be able to deploy an EC2 instance with Lenses installed and connect it to your AWS MSK, Kafka Connect and Schema Registry clusters and create the necessary Security Groups. 

How safe is this?

There are two major elements that make this safe.

When the IAM role is created, it is hardcoded with the Lenses.io AWS Account ID to ensure only Lenses.io can access the role. 

"Principal": {"AWS": "Lenses.io AWS Account ID"},

The IAM role is then also hardcoded with an ExternalID. A unique GUID that represents your personal portal.lenses.io.io account.  This entrusts Lenses to act as a “deputy” that can act on your behalf. It prevents the “confused deputy problem”

"Condition": {"StringEquals": {"sts:ExternalId": "<<SOME GUID UNIQUE TO YOU>>"}}

This ExternalID will be referenced when Lenses makes an AssumeRole API call to request temporary credentials. Only Lenses.io and you (since it’s hard-coded as a Condition in the IAM Role condition) know the ExternalID. This ensures that no other Lenses.io customer can invoke actions against your AWS resources on your behalf. 

This technique avoids the need for Lenses asking for your AWS account access keys and is therefore much safer. 

A full explanation of this practice can be found in AWS’s blog.

How does Lenses secure this information?

Lenses will keep three pieces of information about your environment:

• Your AWS Account ID

• Your AWS Role Name that you created for us

• Your unique External ID

This information is stored encrypted in an Amazon RDS database with keys held in AWS KMS.

Of course, once the stack is created, you are entitled to delete the IAM role yourself should you not want Lenses to access it in future. The benefit of keeping it means you can deploy more stacks in future without needing to recreate a new role each time.

3. Provide the IAM Role to Lenses.io

With the role created from within your AWS Console, all you need to provide Lenses is the Account ID and Role Name from the results of the launched Stack.

4. Provide Connection details to your MSK, Kafka Connect & Schema Registry

Select the region and subnet where you would like to deploy Lenses. The portal will discover any MSK clusters you have deployed in the region simplifying the process further.  Should you have Kafka Connect and Schema Registry running, you can optionally point Lenses to the instances too. 

5. Deploy

That’s it. By creating a workspace, Lenses will be on an EC2 instance in your VPC via Cloudformation template. Whilst it’s deploying, you can check the CloudFormation events in the AWS Console (https://console.aws.amazon.com/cloudformation/home).

The Lenses EC2 AMI is built with OS hardening to be compliant with the DevSec Linux Baseline. Access to the machine must be through EC2 Instance Connect.

Go get up & running on AWS MSK

Get a free two-week trial of Lenses for AWS MSK. No trial key required: Sign up and deploy Lenses here at https://portal.lenses.io/register?version=msk