Data Transformation in TriggerMesh Using MuleSoft DataWeave Scripts

One of the key elements of TriggerMesh is the event transformation engine (code named Bumblebee for obvious reasons). It takes events represented with the CloudEvents specification from a source and modifies them before passing the result to a target. Like all parts of TriggerMesh, this transformation is defined using an API object via a declarative manifest and deployed in Kubernetes. Once applied in TriggerMesh, these transformations can be reused by developers in order to reliably modify events in an expected manner at scale. Because TriggerMesh is built on Knative, these transformations auto-scale out of the box, including scaling down to zero.

Bumblebee transformations are great for JSON to JSON data mapping and there are other transformations available in TriggerMesh like XSLT and XML to JSON.

Some of our customers use MuleSoft® DataWeave and want to transform data Cloud-Natively in Kubernetes while keeping their DataWeave scripts intact. Our new DataWeave Transformation, which is in tech preview and will be open sourced soon, is quite useful if you are trying to bridge legacy integrations in the Cloud era, benefit from a GitOps mindset and migrate to a Kubernetes-based infrastructure while extending the lifetime of your existing investments. The huge benefit is that you can keep your spells intact and create an endpoint that exposes this data transformation without running MuleSoft. You only need TriggerMesh which is open source.

In this blog, we will look at how TriggerMesh can use existing DataWeave scripts to transform your data. We will do this in three steps:

• First, introduce DataWeave
• Second, show how to use the TriggerMesh DataWeave transformation
• Finally, transform events/data

With TriggerMesh, you can transform data Cloud-Natively in Kubernetes while keeping DataWeave scripts intact. This allows you to bridge legacy integrations into the Cloud era, embrace a GitOps mindset, migrate to Kubernetes-based infrastructure, and extend the life of your existing investments. The benefit is that you can keep your spells intact and create an endpoint that exposes this data transformation without running MuleSoft. You only need the open source TriggerMesh.

DataWeave Language

MuleSoft utilizes their own scripting language called DataWeave. This language is a template engine that allows you to transform data to and from any kind of format. Users of MuleSoft who have made mappings in the graphical interface will have DataWeave scripts (called “spells”) that have been generated by these mappings. If you’ve already got these DataWeave spells, you can drop them into your TriggerMesh installation and immediately use them to transform your data.

Here’s an example of a DataWeave spell:

%dw 2.0
output application/json
---
{
 address1: payload.order.buyer.address,
 city: payload.order.buyer.city,
 country: payload.order.buyer.nationality,
 email: payload.order.buyer.email,
 name: payload.order.buyer.name,
 postalCode: payload.order.buyer.postCode,
 stateOrProvince: payload.order.buyer.state
}

This DataWeave spell takes an XML payload, gets details of the “buyer” element in the “order” element, and puts those attributes into a JSON format. The script is expecting the input data to look something like this:

<?xml version='1.0' encoding='UTF-8'?>
<order>
  <product>
    <price>15000</price>
    <model>Extreme Edition</model>
  </product>
  <payment>
    <payment-type>credit-card</payment-type>
    <currency>USD</currency>
  </payment>
  <buyer>
    <email>ace@ventura.com</email>
    <name>Ace</name>
    <address>Koala Boulevard 314</address>
    <city>Beverly Hills</city>
    <state>CA</state>
    <postCode>90210</postCode>
    <nationality>USA</nationality>
  </buyer>
  <accountexec>Marty McFly</accountexec>
</order>

‍

Running DataWeave in TriggerMesh

Now that we have an example DataWeave spell and the expected payload, we can drop the spell into a YAML file to be applied to a TriggerMesh installation. Here’s what that manifest would look like:

apiVersion: components.extensions.triggermesh.io/v1alpha1
kind: DataweaveTransformation
metadata:
  name: dw-xml-to-json
spec:
  dw_spell: |-
        %dw 2.0
        output application/json
        ---
        {
          address1: payload.order.buyer.address,
          city: payload.order.buyer.city,
          country: payload.order.buyer.nationality,
          email: payload.order.buyer.email,
          name: payload.order.buyer.name,
          postalCode: payload.order.buyer.postCode,
         stateOrProvince: payload.order.buyer.state
        }
  output_content_type: application/json
  incoming_content_type: application/xml
  sink:
    ref:
      apiVersion: serving.knative.dev/v1
      kind: Service
      name: sockeye

Of particular interest are the following fields

• kind - This defines the resource type, this is the exciting bit `DataWeaveTransformation`
• name - The name of the transformation, which can be referenced by other TriggerMesh components
• dw_spell - Here’s where we are putting the spell definition that will be run
• output_content_type - Explicitly stating the output type
• incoming_content_type - Explicitly stating the payload type that will be sent in
• sink - This is the TriggerMesh object where the data will be sent

‍

Right now, our sink is sockeye, which is a display for CloudEvents. Here’s how you can run that display:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: sockeye
spec:
  template:
    spec:
      containers:
        - image:docker.io/n3wscott/sockeye:v0.7.0

Note the huge benefit: There is no MuleSoft involved. You just take your spells and put them in a YAML manifest. TriggerMesh fully open source takes over.

‍

Running the Transformation

Once these manifests are applied to your TriggerMesh installation, we can send a test event. First, bring up the Sockeye display by going to the URL found from running:

kubectl get ksvc sockeye -o=jsonpath='{.status.url}'

Then, get the transformation URL:

kubectl get ksvc dw-xml-to-json -o=jsonpath='{.status.url}'

Finally, we can send an XML document via curl to the transformation and see the resulting JSON in Sockeye:

curl --location --request POST 'http://<TRANSFORMATION_URL>' -H 'Content-Type: application/xml' \
--data-raw "<?xml version='1.0' encoding='UTF-8'?>
<order>
  <product>
    <price>15000</price>
    <model>Extreme Edition</model>
  </product>
  <payment>
    <payment-type>credit-card</payment-type>
    <currency>USD</currency>
  </payment>
  <buyer>
    <email>ace@ventura.com</email>
    <name>Ace</name>
    <address>Koala Boulevard 314</address>
    <city>Beverly Hills</city>
    <state>CA</state>
    <postCode>90210</postCode>
    <nationality>USA</nationality>
  </buyer>
  <accountexec>Marty McFly</accountexec>

Conclusion

MuleSoft customers who want to take their DataWeave scripts to a Cloud-Native environment can do so. The TriggerMesh `DataWeaveTransformation` API allows anyone to run data transformation in a Kubernetes cluster. That transformation is available over HTTP(s) by sending a CloudEvent to it, the transformed data is also emitted as a CloudEvent.