Ingesting AWS Events into Google via CloudRun

Ingesting AWS Events into Google via CloudRun
Ingesting AWS Events into Google via CloudRun

AWS has been leading serverless architecture for a while now. After AWS Lambda, their most used service in the serverless category is EventBridge which allows users to build event-driven applications across AWS services and other SaaS applications. Google has announced Eventarc couple years ago which is their version of EventBridge but few event sources were readily available. Only events generated by Google services could be used to trigger Cloud Run. In a recent blog post some new Eventarc features got announced like the ability to trigger Cloud Run services running in Anthos.

Google will need to step up its game in terms of event sources really quick if they want to make a dent in the dominant serverless position of AWS. At TriggerMesh we have developed event sources for some time now. The TriggerMesh platform takes the shape of Kubernetes controller but you can run the event sources as simple containers via docker-compose and of course Cloud Run, which means that you already have non-Google services event sources available. So if you are trying to get AWS events into GCP we got you covered.

You can run TriggerMesh AWS event sources in Google Cloud Run and ingest events from AWS services (e.g SQS, Kinesis, CodeCommit) into Google infrastructure (e.g GKE, Cloud Run). Using Cloud Run you deploy your AWS event consumers in the Google serverless platform with no infrastructure concerns, you just let Cloud Run scale the TriggerMesh containers up and down as needed and ingest the AWS events in the same format as other Google EventArc events: the CNCF cloudevents.

This is useful for anyone who is using AWS services like SQS, or S3 but want to run some processing in Google Cloud, maybe run some ML models. It could also be used to trigger data syncs in real-time, copying files to Google storage as soon as they arrive in S3. While migrating workloads from Cloud to Cloud is not realistic, multi-cloud is indeed real when an enterprise uses both Cloud providers to manage risks or when acquisitions have meant that it has not been possible to stick to a single Cloud.

How it Works

Every TriggerMesh release makes available a container that consumes messages from AWS and emits them in the Cloudevent format.

To run AWS event sources in Cloud Run simply pick the AWS service of interest - say SQS - and run the corresponding container in Cloud Run. For SQS the latest release gives you "gcr.io/triggermesh/awssqssource-adapter:v1.14.0".

To test this you will need to do four things:

  1. Go to Cloud Run and create a service with the container 
  2. Configure a few environment variables so that the source can contact SQS and sent the cloudevents to their destination
  3. Make sure your destination is running
  4. Send messages via the AWS SQS console

Head to the Cloud Run console and create a new service with the TriggerMesh AWS SQS source (aka adapter). Since this event source uses a polling mechanism you need to set its minimum scale to 1, it also does not need to accept all incoming traffic and can be restricted to a specific VPC.

To run properly the source requires a few environment variables, specifically the ARN of the SQS queue, the URL of the destination for the events and a set of AWS API keys that have the proper IAM associated to them in order to read from SQS. The snapshot below shows you the details of the revisions and the environment variables that were set:

Let’s now send a message via the AWS SQS console, this is fairly straightforward and a screenshot is shown below:

With your destination running the source will consume the message and emit a cloudevent towards the destination. In our case we use Sockeye (a cloudevent viewer) deployed in the TriggerMesh cloud. The snapshot below shows you the received event and its attributes.

And voila ! You can see the the Data section that the Body contains the message that you entered in the AWS SQS console.

While the intended use of TriggerMesh is to deploy it in a Kubernetes cluster, we have just shown you that you can use the event sources directly in a serverless offering like Cloud Run. This opens the door to true multi-cloud workflow, consuming events from one Cloud and triggering actions on another in an event driven manner. While you can write those consumers yourself, using TriggerMesh sources saves you from the maintenance of them as well as the learning curve of using the cloudevent formats, you can focus on the real differentiating processes of your business.

Create your first event flow in under 5 minutes