Send Messages to Kafka Declaratively
Blog 2 in our four-part Series: Using TriggerMesh to Connect Apache Kafka And Kubernetes
In blog 1 in this series, we looked at how TriggerMesh lets you declaratively define your Kafka messages and send them in the CloudEvent format to a Kubernetes workload.
In this blog, let’s do the opposite and send messages to a Kafka topic declaratively. This also means that you are able to declaratively define your Kafka sources and sinks with Kubernetes manifests, and avoid the pesky jar compilation and jar uploading to configure Kafka connect.
As described in the blog 1, the goal is to write Kubernetes manifests to declare how we send and receive—or produce and consume—messages to a Kafka topic. The diagram above shows it all. Note that a KafkaSink produces an event to a Kafka topic and therefore is a Kafka source from the point of view of the Kafka cluster. Therefore, the KafkaSource consumes an event from a Kafka topic and is a Kafka sink from the point of view of the Kafka cluster.
Because it would be much easier than writing my own Kafka clients, compiling, packaging, deploying etc. If only I could write a bit of config and let a system like Kubernetes manage it for me that would be great.
We install two controllers and a CRD for a new kind called KafkaSink. For good measure, we also install the knative eventing CRD.
Shortly after the apply, the Pods are running. Note that the sample below also shows the controller for KafkaSource
And our new fancy CRD KafkaSink is in place
The KafkaSink in Action
Using the same secret that allows us to talk to our Kafka cluster in confluent from Section 1, we can now set up a Kafka sink declaratively like so:
What this will do is setup an HTTP endpoint in my Kubernetes cluster where I can POST CloudEvents. These CloudEvents will then be produced to my Kafka cluster. The KafkaSink looks like:
Now that we have an HTTP endpoint acting as an HTTP proxy to a Kafka topic, we can simply POST a CloudEvent to this endpoint and the Sink will “produce” the event to the topic. To do this we can get a quick shell in our cluster and with curl craft a CloudEvent by hand like so:
Checking the logs in the display Pod we see our CloudEvent being received by that KafkaSource that we setup in our previous Blog.
Head over to Blog 3 in this four-part series to learn about configuring Kafka sources and sinks in Kubernetes.