Dagger's approach to building CI/CD pipelines is unique. Dagger SDKs let you develop your pipelines as code, in a programming language you already know, and then execute those pipelines as standard OCI containers. This approach is both supremely portable and consistent (because your pipeline runs the same way locally and in a remote CI environment) and it has the added bonus of compatibility with the existing Docker ecosystem.
One of the most common questions users ask when first encountering Dagger is, "I already have a Dockerfile, why should I use Dagger?" And the answer (like most good answers) depends very much on the intended use case.
Usage Scenarios
It's true that for most small and medium size projects without complex CI/CD needs, a Dockerfile can (and does) get the job done. But there are a number of scenarios where replacing a Dockerfile with a Dagger pipeline can yield significant benefits. Here are a few:
Replacing monolithic CI dependency images: Sometimes, teams maintain one giant everything-including-the-kitchen-sink Dockerfile containing multiple dependencies so they only have to reference one image (like myorg/ci) in all their CI pipelines. Building and pushing these Dockerfiles becomes quite a chore, and teams must also deal with registry rate limits (if using a public registry) or maintain a private registry instead. Dagger lets you maintain images purely as code, so you never have to worry about building and pushing gigantic monolithic CI images.
Sharing data between CI pipelines and application code: Since your Dagger pipeline is usually written in the same language as your application, it's much easier to share data between the two. For example, if you have a constant (like an image name) that you use in your actual production codebase, you can import it directly into your CI code rather than attempting to pass it as a Dockerfile argument.
Greater re-usability and abstraction: Dagger lets you use general purpose programming languages and therefore supports more re-usability and abstraction. You can access native control structures like conditionals and loops, benefit from the language's existing testing tools, and add new functionality or integrate with third-party services using community extensions or packages. These operations are either extremely awkward or unsupported in the standard Dockerfile model, but very easy with Dagger. By the same token, Dagger pipelines also benefit from static typing and easier refactoring.
Gradually rewriting a legacy Dockerfile: Sometimes you are stuck with a Dockerfile that is both very important, and painful to use. Everyone knows it should be replaced with something more powerful, ideally using a real programming language - but rewriting it all at once, without breaking compatibility, is too hard, so it never happens. Since Dagger can natively run Dockerfiles with full compatibility, it’s easy to wrap your existing Dockerfile in a Dagger pipeline, and gradually refactor it over time, without breaking your team’s workflow.
Getting Started
If you've struggled with one or more of these scenarios, or if you're just curious about how it all works, we've put together a brief guide on replacing a Dockerfile with a Dagger pipeline. This guide explains how to use the Dagger Go SDK to perform all the same operations that you would typically perform with a Dockerfile.
It illustrates the process by replacing the Dockerfile for the popular open source Memcached caching system with a Dagger Go pipeline and covers the following common tasks:
Create a Dagger client in Go
Write a Dagger pipeline in Go to:
Configure a container with all required dependencies and environment variables
Download and build the application source code in the container
Set the container entrypoint
Publish the built container image to Docker Hub
Test the Dagger pipeline locally
Introduction
This guide explains how to use the Dagger Go SDK to perform all the same operations that you would typically perform with a Dockerfile, except using Go. You will learn how to:
Create a Dagger client in Go
Write a Dagger pipeline in Go to:
Configure a container with all required dependencies and environment variables
Download and build the application source code in the container
Set the container entrypoint
Publish the built container image to Docker Hub
Test the Dagger pipeline locally
Requirements
This guide assumes that:
You have a Go development environment with Go 1.15 or later. If not, download and install Go.
You have Docker installed and running on the host system. If not, install Docker.
You have a Go module with the Dagger Go SDK installed. If not, install the Dagger Go SDK.
You have a Docker Hub account. If not, register for a Docker Hub account.
Step 1: Understand the source Dockerfile
To illustrate the process, this guide replicates the build process for the popular open source Memcached caching system using Dagger. It uses the Dockerfile and entrypoint script for the official Docker Hub Memcached image.
Begin by reviewing the source Dockerfile and corresponding entrypoint script to understand how it works. This Dockerfile is current at the time of writing and is available under the BSD 3-Clause License.
Broadly, this Dockerfile performs the following steps:
It starts from a base alpine container image.
It adds a memcache user and group with defined IDs.
It sets environment variables for the Memcached version (MEMCACHED_VERSION) and commit hash (MEMCACHED_SHA1).
It installs dependencies in the container.
It downloads the source code archive for the specified version of Memcached, checks the commit hash and extracts the source code into a directory.
It configures, builds, tests and installs Memcached from source using make.
It copies and sets the container entrypoint script.
It configures the image to run as the memcache user.
Step 2: Replicate the Dockerfile using a Dagger pipeline
The Dagger Go SDK enables you to develop a CI/CD pipeline in Go to achieve the same result as using a Dockerfile.
To see how this works, add the following code to your Go module as main.go. Replace the DOCKER-HUB-USERNAME placeholder with your Docker Hub username.
package main
import (
"context"
"fmt"
"os"
"dagger.io/dagger"
)
const (
nproc = "1"
gnuArch = "arm64"
publishAddr = "DOCKER-HUB-USERNAME/my-memcached"
)
func main() {
ctx := context.Background()
// create a Dagger client
client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stdout))
if err != nil {
panic(err)
}
defer client.Close()
// set the base container
// set environment variables
memcached := client.Container().
From("alpine:3.17").
WithExec([]string{"addgroup", "-g", "11211", "memcache"}).
WithExec([]string{"adduser", "-D", "-u", "1121", "-G", "memcache", "memcache"}).
WithExec([]string{"apk", "add", "--no-cache", "libsasl"}).
WithEnvVariable("MEMCACHED_VERSION", "1.6.17").
WithEnvVariable("MEMCACHED_SHA1", "e25639473e15f1bd9516b915fb7e03ab8209030f")
// add dependencies to the container
memcached = setDependencies(memcached)
// add source code to the container
memcached = downloadMemcached(memcached)
// build the application
memcached = buildMemcached(memcached)
// set the container entrypoint
entrypoint := client.Host().Directory(".").File("docker-entrypoint.sh")
memcached = memcached.
WithFile("/usr/local/bin/docker-entrypoint.sh", entrypoint).
WithExec([]string{"ln", "-s", "usr/local/bin/docker-entrypoint.sh", "/entrypoint.sh"}).
WithEntrypoint([]string{"docker-entrypoint.sh"}).
WithUser("memcache")
// publish the container image
addr, err := memcached.Publish(ctx, publishAddr)
if err != nil {
panic(err)
}
fmt.Printf("Published to %s", addr)
}
func setDependencies(container *dagger.Container) *dagger.Container {
return container.
WithExec([]string{
"apk",
"add",
"--no-cache",
"--virtual",
".build-deps",
"ca-certificates",
"coreutils",
"cyrus-sasl-dev",
"gcc",
"libc-dev",
"libevent-dev",
"linux-headers",
"make",
"openssl",
"openssl-dev",
"perl",
"perl-io-socket-ssl",
"perl-utils",
})
}
func downloadMemcached(container *dagger.Container) *dagger.Container {
return container.
WithExec([]string{"sh", "-c", "wget -O memcached.tar.gz https://memcached.org/files/memcached-$MEMCACHED_VERSION.tar.gz"}).
WithExec([]string{"sh", "-c", "echo \"$MEMCACHED_SHA1 memcached.tar.gz\" | sha1sum -c -"}).
WithExec([]string{"mkdir", "-p", "/usr/src/memcached"}).
WithExec([]string{"tar", "-xvf", "memcached.tar.gz", "-C", "/usr/src/memcached", "--strip-components=1"}).
WithExec([]string{"rm", "memcached.tar.gz"})
}
func buildMemcached(container *dagger.Container) *dagger.Container {
return container.
WithWorkdir("/usr/src/memcached").
WithExec([]string{
"./configure",
fmt.Sprintf("--build=%s", gnuArch),
"--enable-extstore",
"--enable-sasl",
"--enable-sasl-pwdb",
"--enable-tls",
}).
WithExec([]string{"make", "-j", nproc}).
WithExec([]string{"make", "test", fmt.Sprintf("PARALLEL=%s", nproc)}).
WithExec([]string{"make", "install"}).
WithWorkdir("/usr/src/memcached").
WithExec([]string{"rm", "-rf", "/usr/src/memcached"}).
WithExec([]string{
"sh",
"-c",
"apk add --no-network --virtual .memcached-rundeps $( scanelf --needed --nobanner --format '%n#p' --recursive /usr/local | tr ',' '\n' | sort -u | awk 'system(\"[ -e /usr/local/lib/\" $1 \" ]\") == 0 { next } { print \"so:\" $1 }')",
}).
WithExec([]string{"apk", "del", "--no-network", ".build-deps"}).
WithExec([]string{"memcached", "-V"})
}DANGER Like the source Dockerfile, this pipeline assumes that the entrypoint script exists in the current working directory on the host as docker-entrypoint.sh. You can either create a custom entrypoint script, or use the entrypoint script from the Docker Hub Memcached image repository.
There's a lot going on here, so let's step through it in detail:
The Go CI pipeline imports the Dagger SDK and defines a main() function. The main() function creates a Dagger client with dagger.Connect(). This client provides an interface for executing commands against the Dagger engine.
It initializes a new container from a base image with the client's Container().From() method and returns a new Container struct. In this case, the base image is the alpine:3.17 image.
It calls the withExec() method to define the adduser, addgroup and apk add commands for execution, and the WithEnvVariable() method to set the MEMCACHED_VERSION and MEMCACHED_SHA1 container environment variables.
It calls a custom setDependencies() function, which internally uses withExec() to define the apk add command that installs all the required dependencies to build and test Memcached in the container.
It calls a custom downloadMemcached() function, which internally uses withExec() to define the wget, tar and related commands required to download, verify and extract the Memcached source code archive in the container at the /usr/src/memcached container path.
It calls a custom buildMemcached() function, which internally uses withExec() to define the configure and make commands required to build, test and install Memcached in the container. The buildMemcached() function also takes care of deleting the source code directory at /usr/src/memcached in the container and executing memcached -V to output the version string to the console.
It updates the container filesystem to include the entrypoint script from the host using withFile() and specifies it as the command to be executed when the container runs using WithEntrypoint().
Finally, it calls the Container.publish() method, which executes the entire pipeline descried above and publishes the resulting container image to Docker Hub.
Step 3: Test the Dagger pipeline
Test the Dagger pipeline as follows:
1. Log in to Docker on the host:
docker loginINFO This step is necessary because Dagger relies on the host's Docker credentials and authorizations when publishing to remote registries.
2. Run the pipeline:
go run main.goDANGER Verify that you have an entrypoint script on the host at ./docker-entrypoint.sh before running the Dagger pipeline.
Dagger performs the operations defined in the pipeline script, logging each operation to the console. This process will take some time. At the end of the process, the built container image is published on Docker Hub and a message similar to the one below appears in the console output:
Published to docker.io/.../my-memcached@sha256:692....Browse to your Docker Hub registry to see the published Memcached container image.
Conclusion
This tutorial introduced you to the Dagger Go SDK. By replacing a Dockerfile with native Go code, it demonstrated how the SDK contains everything you need to develop CI/CD pipelines in Go and run them on any OCI-compatible container runtime.
The advantage of this approach is that it allows you to use all the poweful native language features of Go, such as static typing, concurrency, programming structures such as loops and conditionals, and built-in testing, to create powerful CI/CD tooling for your project or organization.
Use the SDK Reference to learn more about the Dagger Go SDK.
Comments