8 Jul 2016 Read in about 4 min
Important. Our Confluent CSD is deprecated and replaced by our most complete yet solution for a managed Kafka stack through Cloudera Manager, including monitoring, alerts and our exclusive UIs. See it here and request a trial today!
We are happy to announce the first version of our Confluent CSD.
Utilizing Landoop’s Confluent CSD you can create a Kafka Cluster with support services such as REST Proxy, Schema Registry and Kafka Connect in a few clicks.
Cloudera Service Descriptor (CSD) is Cloudera’s technology that enables third parties to extend Cloudera Manager with new services which leverage the features of the platform such as monitoring, configuration, distribution and resource management.
We develop high-quality parcels (CDH’s package format) and a CSD for the Confluent Platform that automates most of the installation and configuration procedure in the hope to make this great technology accessible to more teams.
Once you add the CSD to your Cloudera installation, the procedure to set up a cluster is almost as simple as pressing next -> next -> finish in a common installer.
The platform is centered around Apache Kafka, offering services that extend Kafka with REST endpoints and enable schema enforcement and data validation and interoperability with other components of your stack (e.g RDBMS, Hadoop, APIs).
We may distinguish the following components:
Brokers are the heart of Kafka. They receive and propagate messages, supporting data partitioning and replication.
ZooKeeper is used as a distributed synchronization and configuration store. Given we already have Cloudera’s ZooKeepers, we use them (instead of installing a new ZooKeeper cluster)
Schema Registry is a store for metadata. As the name suggests it is used to store schemas and provides versioned history, compatibility settings and evolution of schemas. Its serializers plug into Kafka clients, turning the messages into structured data and enforcing schemas, making Kafka a lot more powerful.
REST Proxy adds HTTP REST endpoints to Kafka, enabling developers to write Kafka clients and perform administrative tasks without the need of native libraries.
Kafka Connect enables streaming data between Kafka and external data systems, such as RDBMS, HDFS, Hive or REST APIs —an interesting example being the twitter connector which streams tweets to Kafka topics.
Our CSD enables you to add
Confluent Platform 2.0.1 on your CDH cluster. From
Cloudera Manager’s web interface you can add and remove instance roles freely,
as well as perform configuration tasks and basic health and log monitoring.
We took care to add almost all service options into the configuration interface and then some; our goal being to simplify management but not take away options.
Much of our work went into supporting security features of the platform. The platform is still anew, thus lacking in security options but we did our best to support the existing ones. Although it is still work in progress, we do support Kerberos authentication to the brokers and SSL communication with them, yet making easy to maintain unsecure endpoints for the rest of the services. We plan on ACLs support and more security options as the platform evolves.
To install a CSD you only have to add it into a directory in your server, under Cloudera Manager’s installation location, then restart the Cloudera Manager.
Usually the location is
/opt/cloudera/csd. The most common procedure for a
modern Linux distribution would be to ssh into your Cloudera Manager server and
$ sudo mkdir -p /opt/cloudera/csd $ sudo wget "https://archive.landoop.com/cdh5/csd/Confluent-Platform-2.0.1-0.jar" \ -O "/opt/cloudera/csd/Confluent-Platform-2.0.1.jar" $ sudo chown -R cloudera-scm:cloudera-scm /opt/cloudera/csd $ sudo systemctl restart cloudera-scm-server
Update 17th August 2016: we recommended the newer 3.0.0 version
Once you login to Cloudera Manager you will be able to install the Confluent
Parcel from your parcels’ page. This will add Confluent’s distribution to your
cluster and make its tools available under
/usr/bin. Please note that
Confluent Parcel is incompatible with Cloudera’s Kafka Parcel since their
functionality overlap. If you have a Kafka installation in your cluster please
remove it before installing our parcel. Once installed, our parcel will provide
you with all the Kafka tools, as they are provided by the distribution of the
Confluent Platform. Should you ever decide to remove the Confluent Platform, you
can re-install your old Kafka installation if desired.
Once the parcel is installed you can proceed to add a Confluent Platform service to your cluster. No special settings are needed; assigning roles should be enough to get you started.
If you need help head over to our installation docs. Our documentation also covers more complex scenarios, like Kerberos setups.
For commercial support plans please contact us. Support packages - 35 $ / mo per server paid annually.
We are already working on Confluent’s latest release, 3.0.0, and we plan to implement and support more security features. We perform rigorous testing to our releases –we do use them in Landoop after all– and this takes time.
Thank you for your interest !
Did you like this article?
Industrialize your Kafka with Lenses