My First Golang

I've written a fair amount of Golang, but until recently, none of it made its way into a production.

In fact, I've thrown away most of the Go code I've written. Some was part of side projects that went nowhere. Some were just practice (getting used to concurrency, channels, error handling, etc).

Why Throw it Away

The more interesting cases of thrown-away Go was because many times it ended up being simpler staying within the same stack that everything else was written in.

For me, this stack is almost always PHP/Laravel.

Dropping Golang in favor of keeping with PHP (both in Chipper CI and elsewhere) wasn't necessarily simpler because Golang was complex (altho it can be). It's just that staying in the stack the whole team is familiar with is often a better idea.

When I Use PHP

I don't really like using Go for HTTP handling. Unless you have specific performance needs, I still think PHP is best suited for application logic in the web layer.

Specifically, Laravel has always seemed best-in-class for the standard web-app stuff (forms, validation, storing data) and still shines for more complex stuff (Chipper CI is running Docker-based builds with PHP).

Additionally, PHP fits the HTTP lifecycle much better than most other languages. It was always meant to be born, live, and die within each HTTP request. Just as HTTP is a stateless protocol, PHP is "stateless" in that it builds it's entire universe within each request. This takes away a lot of worry you may have around running apps as long-running processes (like many/most other languages).

When I Use Golang

I really enjoy Golang for utitiles and "plumbing". What I like best is:

1. Performance that you'd expect from a compiled language
2. Concurrency is easy-ish (if you need it)
3. Ease of deployment
4. It's not an overly complex language (in theory - it does seem easy to write overly complicated code)
5. Most importantly: It does async/concurrency, which PHP sorely lacks (the open source tools for async in PHP are too cumbersome for our use case in my opinion).

Chipper CI's First Golang

As outlined in the article Our Naive Build System we outlined how we use PHP to run docker-compose commands to run builds within Docker containers.

One detail of this is that the build container just ran tail -f /dev/null to keep the container alive while we executed the pipeline scripts.

It had some issues - notably, some containers never died/were cleaned up! I wanted something that worked a bit better. I had two goals:

1. A long-running process whose purpose is to keep the container alive while we exec pipeline scripts within the container
2. Enforce the 1 hour build timelimit (which was not enforced within the container before).

Golang was well suited for this:

1. Easy to build into the deploy container
2. You create just one binary with no other dependencies (unlike, say, NodeJS)
3. I have future plans for more idea that Golang is well suited for

The Golang

It's fairly simple.

package main

import (
    "context"
    "log"
    "os"
    "os/signal"
    "syscall"
    "time"
)

func main() {
    pipelineTimeout, cancel := context.WithTimeout(context.Background(), time.Hour)
    defer cancel()

    signals := make(chan os.Signal, 1)
    done := make(chan bool, 1)

    signal.Notify(signals, syscall.SIGINT, syscall.SIGTERM)

    go func(sig <-chan os.Signal, fin chan<-bool) {
        <-sig
        fin <- true
    }(signals, done)

    loop:
    for {
        select {
        case <-done:
            break loop
        case <-pipelineTimeout.Done():
            log.Fatalf("Pipeline has reached it's max execution time")
        }
    }

    os.Exit(0)
}

We use context.WithTimeout() and have it start counting down an hour. If the hour is hit, the process end with a fatal error (if you noticed, I don't worry about exiting vs using panic() to allow the defered function call to run).

Then we listen for some signals, specifically SIGINT and SIGTERM, both of which may be sent by Docker when telling a container to stop.

We make use of some channels and a go func to push listening for signals to the background in a goroutine.

We use a for loop with a select (a common Golang convention) to wait for either a signal to tell us we're done, or the 1 hour timeout to hit.

This is, in theory, simple Golang code. However if you're not familar with some Go conventions (for loop + select), or channels, or the use of "go" for concurrency, then it may actually be confusing as all hell. Which is also a gripe I have with Golang in general :D

I'm looking at the code just before publishing this and I see all sorts of subtle "issues". For example, the deferred cancel() function is likely never called since I either call os.Exit() or log.Fatalf() (where as a panic() would allow deferred functions to run). But I want to be sure to return a 0 exit code, and I don't want to use panic as it's not appropriate to spit out a stack trace in this situation.

My use of channels feels wonky. In fact, I'm not sure I even need to listen for signals to return a 0 exit code, except the way I tested suggested I might.

I probably could skip naming the loop and breaking out of it by just running os.Exit(0) within the select. But I was afraid of some strange conditon where the loop was broken from and the script reached the end (dumb!).

There's all sorts of considerations that cause a lot of uncertainty! In fact, I'll probably rewrite this thing :D

The Build Container

Our build container uses multi-stage builds build the golang utility and get it into the main build container.

###
# Build the chipper command
##
FROM golang:1.14 as chipper-builder

WORKDIR /go/src/github.com/chipperci/chipper/
COPY chipper.go .
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o chipper .


###
# Build the build container
##
FROM ubuntu:18.04 as chipper

COPY --from=chipper-builder /go/src/github.com/chipperci/chipper/chipper /usr/local/bin/chipper

# stuff omitted...

As mentioned above, this builds the chipper command in one container and copies it into the build container so we can use it!

The Future

This is a very small start. We have plans to expand the use of Golang to help us run commands and stream output in a more performant way (something Golang is very well suited for).

In particular, we're excited to implement gRPC to help us stream script output over HTTP/2. I've come to like gRPC over a REST(ish) api or something even more complex. While the docs aren't great, grabbing a $12 course on Udemy to learn about gRPC was really useful to get me up and running.

The code-generation to build out your API from a proto file is something I found myself liking more than I thought I would!

Code Update

As I hinted at, I rewrote this code. And it's much simpler!

I the end, I didn't need to ensure the script has an exit code of 0 for its use case.

Unix convention is that an exit code on a signal = 128 + signal value, so a SIGTERM (value 15) results in code 128 + 15 = 143 and SIGINT (value 2) results in exit code 128 + 2 = 130.

So the Go code now just needs to care about the 1 hour time limit:

package main

import (
    "context"
    "log"
    "time"
)

func main() {
    pipelineTimeout, _ := context.WithTimeout(context.Background(), time.Hour)

    <-pipelineTimeout.Done()
    log.Fatal("Pipeline has reached its max execution time")
}

We can ignore the cancel() function (via the use of _) returned by the timeout context since we exit the program completely on a signal sent by Docker (telling the container process to stop) or the timeout is reached.

Since we aren't listening for more than one channel (and since the channel given to us by context.WithTimeout is blocking) we no longer need the for loop + switch.

Much simpler!