Writing a Docker Log Driver plugin — Basics

Docker Logging Basics

When a program , likely a server is run within a docker container, you can docker logs command to view the program output as written to its STDOUT.
At a high level what is happening here is that docker consults a logging driver for the container which returns the “captured” STDOUT of your program. Each Docker daemon has a default logging driver, which each container uses, unless, you configure it to use a different logging driver. In addition to using the logging drivers included with Docker, you can also implement and use logging driver plugins.

Logging Driver Plugin

Why would you want to write a plugin?

For custom storage and processing of logs. For example, say you wanted to stream the programs STDOUT to an HTTP Service where you can later analyze its output. BTW, this is what Splunk driver does , sending program STDOUT to Splunk server

I was curious to see what it will take to get a driver up and running but couldn’t find any comprehensive resource from docker that describes it in detail. After some research I was able to write and run a simple driver .

In this article I will share what is needed to write, configure, install and run a logging driver. The code and references for this article can be found at the same GitHub repository — https://github.com/monmohan/logdriver

The Contract

A Log Driver plugin needs to handle following HTTP Requests —

Start Logging Request

Called when the container is started with the configured logging driver. There are couple of key things to understand about this request. As part of this HTTP request, Docker includes the ID of the container and also a handle to a named pipe for that container. STDOUT from the programs in the container is available on this FIFO and hence Driver should open this FIFO as a reader to continuously ingest the program output.

Stop Logging Request

Called when container is stopped. Gives Driver a chance to cleanup the resources allocated for reading the FIFO stream.

Get Supported Capabilities

Drivers can indicate whether they support ability to read logs (called when docker logs is invoked). For example, It may not make sense for a logging driver which ships log to remote location, to support this functionality

Read Logs

docker logs invokes this call on the driver, to retrieve logs.

Request body are in JSON format. I will not go into the detail of all the Request format. Please check out http.go here , for the request/response details

How does Docker call these HTTP Requests?

In short, over a unix socket. The socket name is a setting in the driver configuration, see config.json

“interface”: {
“types”: [“docker.logdriver/1.0”],
“socket”: “mylogdriver.sock”
},

Docker provides helpers to write Plugins and we make use of the helper to setup our HTTP Handler. See https://github.com/docker/go-plugins-helpers/blob/master/sdk/handler.go#L58

ServeUnix makes the handler listen for requests in a unix socket. It also creates the socket file in the right directory for docker to read.

Implementing the Driver

Few things to keep in mind —

• Same driver process gets “start logging” request for multiple containers on the same host. This means that the driver should keep track of which FIFO is linked to which container, in order to separate those logs. For example by keeping a map of containerID to FIFO stream.
• The FIFO stream is protobuf encoded so it needs to be decoded in order to read the actual log message
• The driver process can’t block when handing the start logging request, so a natural way to process, would be starting a Goroutine which ingests from the FIFO .
• For more details checkout driver source code

Configure and Install the Driver

Plugins are essentially docker images. So in order to create a plugin,

1. first we build a standard docker image with all the driver and HTTP code.
2. Then using the docker create command we create a write-able container layer over the docker image we created.
3. Exporting this new created container to a directory would give you the container rootfs.
4. Plugin config (config.json) can then be added to this container rootfs
5. A fully functional plugin image can then be created from this rootfs.
6. See https://github.com/monmohan/logdriver/blob/master/plugin/Makefile for details of the build and enable process.

Next

In this article we saw what it takes to build a working log driver. We will see our log driver in action in the next article . Stay tuned !