Pogorelovs

Navigate back to the homepage

Playing with MSK kafka

Sergey Pogorelov
February 4th, 2020 · 1 min read

When you work with MSK you might want to connect with local producer/consumer to it for the test purposes, but you should keep in mind that MSK (as all AWS services) requires connection through SSL and here you’ll need to specify it in your Kafka client.

In order to connect to your msk with local consumer/producer:

  1. Download Kafka binaries and extract them
  2. Copy cacerts from your Java dir
    1cp /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/security/cacerts \
    2~/software/kafka.client.truststore.jks
  3. Create client.properties with following contents and point it to the file that you’ve copied in step 2:
    1security.protocol=SSL
    2ssl.truststore.location=/home/ubuntu/software/kafka_2.11-2.4.0/bin/kafka.client.truststore.jks
  4. Now you can run your consumer/producer

Running the Consumer

1$KAFKA_HOME/bin/kafka-console-consumer.sh \
2--bootstrap-server $BOOTSTRAP_SERVERS \
3--consumer.config client.properties \
4--topic $TOPIC_NAME

Running the Producer

1$KAFKA_HOME/bin/kafka-console-producer.sh \
2--broker-list $BOOTSTRAP_SERVERS \
3--producer.config client.properties \
4--topic $TOPIC_NAME

Getting the bootstrap servers

You can get the $BOOTSTRAP_SERVERS via the aws-cli:

1aws kafka get-bootstrap-brokers --cluster-arn $ClusterArn

In order to get the ClusterArn you might need to list the available MSK clusters and get their details

1aws emr list-clusters --region $REGION
2aws emr describe-cluster --region $REGION --cluster-id $CLUSTER_ID

Keep in mind that you’ll need the BootstrapBrokerStringTls (usually ends with port 9094) in order to connect through SSL.

Now you can consume and produce message via the local Kafka binaries to you MSK cluster.

Join my email list and get notified about new content

Be the first to receive my latest content with the ability to opt-out at anytime. I promise not to spam your inbox or share your email with any third parties.

More articles from Pogorelovs

Simple local Nginx

It's preety simple to run a local Nginx web server for test purposes...

February 4th, 2020 · 1 min read

OS X environment setup guide

When I used Ubuntu as my main system I experimented a lot and often reinstalled the system. I’ve got to the conclusion that it might be handy to have a list of steps to do after a fresh system install...

December 7th, 2018 · 1 min read
© 2018–2020 Pogorelovs
Link to $https://github.com/sergeioffLink to $https://www.linkedin.com/in/sergeioff