Blog 2 in our four-part Series: Using TriggerMesh to Connect Apache Kafka And Kubernetes
In blog 1 in this series, we looked at how TriggerMesh lets you declaratively define your Kafka messages and send them in the CloudEvent format to a Kubernetes workload.
In this blog, let’s do the opposite and send messages to a Kafka topic declaratively. This also means that you are able to declaratively define your Kafka sources and sinks with Kubernetes manifests, and avoid the pesky jar compilation and jar uploading to configure Kafka connect.
As described in the blog 1, the goal is to write Kubernetes manifests to declare how we send and receive—or produce and consume—messages to a Kafka topic. The diagram above shows it all. Note that a KafkaSink produces an event to a Kafka topic and therefore is a Kafka source from the point of view of the Kafka cluster. Therefore, the KafkaSource consumes an event from a Kafka topic and is a Kafka sink from the point of view of the Kafka cluster.
Because it would be much easier than writing my own Kafka clients, compiling, packaging, deploying etc. If only I could write a bit of config and let a system like Kubernetes manage it for me that would be great.
We install two controllers and a CRD for a new kind called KafkaSink. For good measure, we also install the knative eventing CRD.
What this will do is setup an HTTP endpoint in my Kubernetes cluster where I can POST CloudEvents. These CloudEvents will then be produced to my Kafka cluster. The KafkaSink looks like:
kubectl get kafkasink
NAME URL AGE READY REASON
Now that we have an HTTP endpoint acting as an HTTP proxy to a Kafka topic, we can simply POST a CloudEvent to this endpoint and the Sink will “produce” the event to the topic. To do this we can get a quick shell in our cluster and with curl craft a CloudEvent by hand like so:
Sebastien is a co-founder of TriggerMesh. He has been involved with open source for almost 20 years. Sebastien is an Apache Software Foundation committer and an early contributor to the Kubernetes ecosystem. Around 2014, he became fascinated by the meteoric rise of Docker and decided to learn as much as he could, which led him to author the O’Reilly Docker cookbook. It was this experience that introduced him to Kubernetes, and it was love at first Pod. Sebastien became an early and active participant in the CNCF space, contributing, building, teaching, and innovating. He founded TriggerMesh to push the boundaries of the serverless space and allow all companies to benefit from this new paradigm, even those with on-premises applications.