Kafka

This guide will help you to install the Kafka service on your DCOS cluster backed by Portworx volumes for persistent storage.

The source code for these services can be found here: Portworx DCOS-Commons Frameworks

Please make sure you have installed Portworx on DCOS before proceeding further.

Finding the service in the DCOS Universe/Catalog

You should see the Kafka service available in your universe as Portworx-Kafka

Installation

Default Install

If you want to use the defaults, you can now run the dcos command to install the service

 dcos package install --yes portworx-kafka

You can also click on the “Install” button on the WebUI next to the service and then click “Install Package”.

Advanced Install

If you want to modify the default, click on the “Install” button next to the package on the DCOS UI and then click on “Advanced Installation”

Here you have the option to change the service name, volume name, volume size, and provide any additional options that you want to pass to the docker volume driver. You can also configure other Kafka related parameters on this page including the number of broker nodes.

Click on “Review and Install” and then “Install” to start the installation of the service.

Install Status

Once you have started the install you can go to the Services page to monitor the status of the installation.

If you click on the Kafka service you should be able to look at the status of the nodes being created.

When the Scheduler service as well as all the Kafka services are in Running (green) status, you should be ready to start using the Kafka service.

If you check your Portworx cluster, you should see multiple volumes that were automatically created using the options provided during install, one for each of the Brokers.

If you run the “dcos service” command you should see the portworx-kafka service in ACTIVE state with 3 running tasks

 dcos service

 NAME                           HOST             ACTIVE  TASKS  CPU   MEM      DISK   ID
 portworx-kafka      ip-10-0-3-116.ec2.internal   True     3    3.0  6144.0    0.0    66d598b0-2f90-4d0a-9567-8468a9979190-0038
 marathon                    10.0.7.49            True     2    2.0  2048.0    0.0    66d598b0-2f90-4d0a-9567-8468a9979190-0001
 metronome                   10.0.7.49            True     0    0.0   0.0      0.0    66d598b0-2f90-4d0a-9567-8468a9979190-0000
 portworx            ip-10-0-1-127.ec2.internal   True     4    3.3  4096.0    25.0   66d598b0-2f90-4d0a-9567-8468a9979190-0031
 portworx            ip-10-0-2-42.ec2.internal    True     3    1.2  3168.0  12288.0  66d598b0-2f90-4d0a-9567-8468a9979190-0032

Verify Setup

From the DCOS client; install the new command for portworx-kafka

  dcos package install portworx-kafka --cli

Find out all the kafka broker endpoints

  dcos portworx-kafka endpoints broker

  {
   "address": [
    "10.0.2.82:1025",
    "10.0.0.49:1025",
    "10.0.3.101:1029"
   ],
  "dns": [
  "kafka-2-broker.portworx-kafka.mesos:1025",
  "kafka-0-broker.portworx-kafka.mesos:1025",
  "kafka-1-broker.portworx-kafka.mesos:1029"
   ],
  "vip": "broker.portworx-kafka.l4lb.thisdcos.directory:9092"
  }

Find out the zookeeper endpoint for the created kafka service

 dcos portworx-kafka endpoints zookeeper

 master.mesos:2181/dcos-service-portworx-kafka

Create a topic, from the DCOS client use dcos command to create a test topic test-one with replication set to three

dcos portworx-kafka topic create test-one --partitions 1 --replication 3

{
    "message": "Output: Created topic \"test-one\".\n"
}

Connect to the master node:

 dcos node ssh --master-proxy --leader

and launch a kafka client container:

docker run -it mesosphere/kafka-client

 root@d19258d46fd3:/bin#

Produce a message and send it to all kafka brokers:

 echo "Hello, World." | ./kafka-console-producer.sh --broker-list kafka-2-broker.portworx-kafka.mesos:1025,kafka-0-broker.portworx-kafka.mesos:1025,kafka-1-broker.portworx-kafka.mesos:1029 --topic test-one

Consume the message

 ./kafka-console-consumer.sh --zookeeper master.mesos:2181/dcos-service-portworx-kafka --topic test-one --from-beginning

 Hello, World.