Accessing Kafka on host machine from minikube pods
.. well this applies to accessing pretty much anything on your host machine. There are times when you want to access processes running on your host machine e.g. databases etc. from your minikube
Kubernetes
cluster in Virtual Box (or any other supported provider)
Mind you, this is not fancy at all. In fact, there are better ways e.g. simply using Kubernetes
which comes along with Docker for Mac (but not everyone is using a Mac).
Anyway, this worked well for me….
ngrok
to the rescue!
It’s as simple as …
- setting up
ngrok
- sign up for a (free) account and get your auth token
- start
Kafka
on your host machine, and just type
ngrok tcp 9092(assuming Kafka is listening on default port 9092)
This will start forwarding requests to your local Kafka cluster on port 9092
. you should see something similar to this — notice the highlighted URL
Session Status online
Account Abhishek Gupta (Plan: Free)
Version 2.2.8
Region United States (us)
Web Interface http://127.0.0.1:4040
Forwarding tcp://0.tcp.ngrok.io:16387 -> localhost:9092
Try it out
Create a topic
./kafka-topics.sh --create --topic test --partitions 1 --replication-factor 1 --zookeeper localhost:2181
Produce some data
./kafka-console-producer --broker-list 0.tcp.ngrok.io:16387 --property "parse.key=true" --property "key.separator=:" --topic test//prompt for data>hi:there
>ola:amigo
Make sure that you
- use the above URL for
--broker-list
and - use the correct topic name (one you created in the previous step)
- and enter data when prompted..
Start consuming
./kafka-console-consumer --bootstrap-server 0.tcp.ngrok.io:16387 --topic test --property print.key=true --from-beginning
Again, make sure
- use the above URL for
--bootstrap-server
and - use the correct topic name (one you created in the previous step)
You should see the data coming through.. :-)
Short and sweet. Stay tuned for more
Cheers!