When you work with MSK you might want to connect with local producer/consumer to it for the test purposes, but you should keep in mind that MSK (as all AWS services) requires connection through SSL and here you’ll need to specify it in your Kafka client.
In order to connect to your msk with local consumer/producer:
- Download Kafka binaries and extract them
- Copy cacerts from your Java dir1cp /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/security/cacerts \2~/software/kafka.client.truststore.jks
client.propertieswith following contents and point it to the file that you’ve copied in step 2:1security.protocol=SSL2ssl.truststore.location=/home/ubuntu/software/kafka_2.11-2.4.0/bin/kafka.client.truststore.jks
- Now you can run your consumer/producer
Running the Consumer
1$KAFKA_HOME/bin/kafka-console-consumer.sh \2--bootstrap-server $BOOTSTRAP_SERVERS \3--consumer.config client.properties \4--topic $TOPIC_NAME
Running the Producer
1$KAFKA_HOME/bin/kafka-console-producer.sh \2--broker-list $BOOTSTRAP_SERVERS \3--producer.config client.properties \4--topic $TOPIC_NAME
Getting the bootstrap servers
You can get the
$BOOTSTRAP_SERVERS via the aws-cli:
1aws kafka get-bootstrap-brokers --cluster-arn $ClusterArn
In order to get the ClusterArn you might need to list the available MSK clusters and get their details
1aws emr list-clusters --region $REGION2aws emr describe-cluster --region $REGION --cluster-id $CLUSTER_ID
Keep in mind that you’ll need the BootstrapBrokerStringTls (usually ends with port 9094) in order to connect through SSL.
Now you can consume and produce message via the local Kafka binaries to you MSK cluster.