Package Kubernetes Manifests in a Container Image with Carvel

Package Kubernetes Manifests in a Container Image with Carvel
Package Kubernetes Manifests in a Container Image with Carvel

This post is pretty geeky so be warned. This is the first installment in looking at the tools from VMware Carvel ecosystem. My ultimate interest is in talking about kapp and the kapp-controller because they look a bit like TriggerFlow, the internal brain that helps us control integrations within TriggerMesh. It is an Application CRD inspired application packager/deployer/statuserer…

What is fascinating about the Carvel tools is their simplicity and the single mindedness of each tool, as well as the clear potential alternative to a Helm centric developer ecosystem. Surely the future of Tanzu tooling will be based on Carvel, so VMware users should get familiar with them and the rest of Kubernetes users can definitely be inspired and use them as well.

Before I talk about kapp I need to do a detour via imgpkg. So here it is

OCI and When Images Can Store Anything

When alternative container runtimes started to appear (i.e runc ) and that security around container images distribution became much needed, CoreOS helped to push the entire industry players to sit around the table and create the Open Container Initiative (OCI). What came out of this is two specifications. One for container runtime and one for container images. Shortly after everything started becoming “OCI compliant”. One last shoe to drop sort of speak was the development of OCI artifacts to store pretty much anything using the same API as OCI container images.

Since Helm had become the leading “package” manager for Kubernetes, one of the first OCI artifacts that appeared was about encapsulating Charts into OCI images. This opened the door to more secure Chart manipulation. This also started a new mindset which is that an OCI image can contain random files that you may need to define your app.

You can package Kubernetes manifests in an OCI image and store them in a OCI compliant registry

So how do you do this ? The simplest tool to use is imgpkg

imgpkg from Carvel to the rescue

imgpkg is a standalone tool in the Carvel tool suite, the core scenario is:

You want to create an immutable artifact containing Kubernetes configuration and images used in that configuration. Later, you want to grab that artifact and deploy it to Kubernetes.

We will see in a later post how you can do this end to end packaging and deployment using kapp or the kapp-controller but for now, let’s store a Kubernetes manifest in a OCI image with imgpkg

First, download the binary from the GitHub release page and make sure it is in your path.

$ which imgpkg‍
$ imgpkg -h‍
imgpkg stores files as Docker images (copy, pull, push, tag, version)
imgpkg [command]...

Create a small Pod manifest

kubectl run nginx --image=nginx --dry-run=client -o yaml > pod.yaml

And now use imgpkg to define a new container image (e.g

$ imgpkg push -i -f pod.yaml
file: pod.yaml
Pushed ‘’

That’s it, you just stored some random YAML into an OCI compliant image and pushed it to Google Container Registry. While the interesting part will be to deploy this manifest and ultimately doing it GitOps style in this post we can go a tiny bit further and deconstruct this image to see a bit more clearly.

Deconstruct the Image with crane

Google has also developed its own developer tooling around Kubernetes, things like kustomize of course and skaffold come to mind. Crane is a little heralded tool buried in the depth of GitHub and unmarketed but so useful even if solely for understanding purposes.

With crane you can manipulate container images, pull, push them, copy files etc. So let’s pull our OCI image built with imgpkg

crane pull pod.tar

And what do we see in this tarball ? We see the beautiful structure of an OCI image

$ tar -tvf pod.tar
-rw-r — r — 0 0 0 273 Jan 1 1970 sha256:4e00e3ce89a94eb151f69ff6c9c62fdd31fab493319f1fc23958b58fdf96c383-rw-r
— r — 0 0 0 252 Jan 1 1970 de97fee1d021d94830ab1e46188ab1c1cab4a877de139e3037a49359d2587c93.tar.gz-rw-r
— r — 0 0 0 216 Jan 1 1970 manifest.json

A manifest.json file which references a config file and a single layer. Sadly the manifest does not set a mediaType to tell us what type of content is stored in this “image”.

“Config”: “sha256:4e00e3ce89a94eb151f69ff6c9c62fdd31fab493319f1fc23958b58fdf96c383”,
“RepoTags”: [
“Layers”: [ “de97fee1d021d94830ab1e46188ab1c1cab4a877de139e3037a49359d2587c93.tar.gz”

And the layer simply contains our Pod manifest

$ tar -zxvf de97fee1d021d94830ab1e46188ab1c1cab4a877de139e3037a49359d2587c93.tar.gzx

That’s it! Stay tune for a deeper look at Carvel, there is ytt there is kbld and of course kapp which I already mentioned. All these tools are single minded embrace immutability and a declarative mindset while leaving you the choice in the way you want to do things.

Create your first event flow in under 5 minutes