By Marios Andreopoulos, 8 Jul 2016


Fast Data CSD

Important. Our Confluent CSD is deprecated and replaced by our most complete yet solution for a managed Kafka stack through Cloudera Manager, including monitoring, alerts and our exclusive UIs. See it here and request a trial today!

We are happy to announce the first version of our Confluent CSD.

Utilizing Landoop’s Confluent CSD you can create a Kafka Cluster with support services such as REST Proxy, Schema Registry and Kafka Connect in a few clicks.

Cloudera Service Descriptor (CSD) is Cloudera’s technology that enables third parties to extend Cloudera Manager with new services which leverage the features of the platform such as monitoring, configuration, distribution and resource management.

We develop high-quality parcels (CDH’s package format) and a CSD for the Confluent Platform that automates most of the installation and configuration procedure in the hope to make this great technology accessible to more teams.

If you are sold already, grab the latest stable CSD and head over to our installation docs.

Once you add the CSD to your Cloudera installation, the procedure to set up a cluster is almost as simple as pressing next -> next -> finish in a common installer.

Confluent Platform

The platform is centered around Apache Kafka, offering services that extend Kafka with REST endpoints and enable schema enforcement and data validation and interoperability with other components of your stack (e.g RDBMS, Hadoop, APIs).

We may distinguish the following components:

  • Kafka Brokers
  • ZooKeeper
  • Schema Registry
  • Kafka Rest Proxy
  • Kafka Connect

Brokers are the heart of Kafka. They receive and propagate messages, supporting data partitioning and replication.

ZooKeeper is used as a distributed synchronization and configuration store. Given we already have Cloudera’s ZooKeepers, we use them (instead of installing a new ZooKeeper cluster)

Schema Registry is a store for metadata. As the name suggests it is used to store schemas and provides versioned history, compatibility settings and evolution of schemas. Its serializers plug into Kafka clients, turning the messages into structured data and enforcing schemas, making Kafka a lot more powerful.

REST Proxy adds HTTP REST endpoints to Kafka, enabling developers to write Kafka clients and perform administrative tasks without the need of native libraries.

Kafka Connect enables streaming data between Kafka and external data systems, such as RDBMS, HDFS, Hive or REST APIs —an interesting example being the twitter connector which streams tweets to Kafka topics.

Landoop Confluent CSD

Our CSD enables you to add Confluent Platform 2.0.1 on your CDH cluster. From Cloudera Manager’s web interface you can add and remove instance roles freely, as well as perform configuration tasks and basic health and log monitoring.

We took care to add almost all service options into the configuration interface and then some; our goal being to simplify management but not take away options.

Much of our work went into supporting security features of the platform. The platform is still anew, thus lacking in security options but we did our best to support the existing ones. Although it is still work in progress, we do support Kerberos authentication to the brokers and SSL communication with them, yet making easy to maintain unsecure endpoints for the rest of the services. We plan on ACLs support and more security options as the platform evolves.

Quick install

To install a CSD you only have to add it into a directory in your server, under Cloudera Manager’s installation location, then restart the Cloudera Manager.

Usually the location is /opt/cloudera/csd. The most common procedure for a modern Linux distribution would be to ssh into your Cloudera Manager server and run:

$ sudo mkdir -p /opt/cloudera/csd
$ sudo wget "" \
            -O "/opt/cloudera/csd/Confluent-Platform-2.0.1.jar"
$ sudo chown -R cloudera-scm:cloudera-scm /opt/cloudera/csd
$ sudo systemctl restart cloudera-scm-server

Update 17th August 2016: we recommended the newer 3.0.0 version

Once you login to Cloudera Manager you will be able to install the Confluent Parcel from your parcels’ page. This will add Confluent’s distribution to your cluster and make its tools available under /usr/bin. Please note that Confluent Parcel is incompatible with Cloudera’s Kafka Parcel since their functionality overlap. If you have a Kafka installation in your cluster please remove it before installing our parcel. Once installed, our parcel will provide you with all the Kafka tools, as they are provided by the distribution of the Confluent Platform. Should you ever decide to remove the Confluent Platform, you can re-install your old Kafka installation if desired.

Once the parcel is installed you can proceed to add a Confluent Platform service to your cluster. No special settings are needed; assigning roles should be enough to get you started.

If you need help head over to our installation docs. Our documentation also covers more complex scenarios, like Kerberos setups.

For commercial support plans please contact us. Support packages - 35 $ / mo per server paid annually.

The future

We are already working on Confluent’s latest release, 3.0.0, and we plan to implement and support more security features. We perform rigorous testing to our releases –we do use them in Landoop after all– and this takes time.

Thank you for your interest !

Landoop team.

Related Blogs

Ready to get started with Lenses?

Download free version