Playing with MSK kafka

When you work with MSK you might want to connect with local producer/consumer to it for the test purposes, but you should keep in mind that MSK (as all AWS services) requires connection through SSL and here you’ll need to specify it in your Kafka client.

In order to connect to your msk with local consumer/producer:

1. Download Kafka binaries and extract them
2. Copy cacerts from your Java dir

   Copy 
   1cp /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/security/cacerts \
   2~/software/kafka.client.truststore.jks

3. Create client.properties with following contents and point it to the file that you’ve copied in step 2:

   Copy 
   1security.protocol=SSL
   2ssl.truststore.location=/home/ubuntu/software/kafka_2.11-2.4.0/bin/kafka.client.truststore.jks

4. Now you can run your consumer/producer

Running the Consumer

Copy 
1$KAFKA_HOME/bin/kafka-console-consumer.sh \
2--bootstrap-server $BOOTSTRAP_SERVERS \
3--consumer.config client.properties \
4--topic $TOPIC_NAME

Running the Producer

Copy 
1$KAFKA_HOME/bin/kafka-console-producer.sh \
2--broker-list $BOOTSTRAP_SERVERS \
3--producer.config client.properties \
4--topic $TOPIC_NAME

Getting the bootstrap servers

You can get the $BOOTSTRAP_SERVERS via the aws-cli:

Copy 
1aws kafka get-bootstrap-brokers --cluster-arn $ClusterArn

In order to get the ClusterArn you might need to list the available MSK clusters and get their details

Copy 
1aws emr list-clusters --region $REGION
2aws emr describe-cluster --region $REGION --cluster-id $CLUSTER_ID

Keep in mind that you’ll need the BootstrapBrokerStringTls (usually ends with port 9094) in order to connect through SSL.

Now you can consume and produce message via the local Kafka binaries to you MSK cluster.