Dev Environments Within Docker Containers
Introduction
You’re a software engineer with a new laptop.
You’re going to be writing code in multiple languages.
You’re going to have projects of varying dependencies.
But you want to avoid the issues you’ve had in the past:
- Messy installations of lots of different software packages.
- Clogging up your system with programming language version managers.
You decide to use Docker.
It’s a dependency sure, but you’ve got to install something!
With Docker it allows your environment to stay clean.
So let’s see two simple examples:
Note: I don’t claim this to be ‘the way’ to do this.
This is just a setup that works well enough for me.
I’m also a Vim user, so your mileage may vary.
Python
You have a Python project you need to work on.
I won’t explain how Python works,
I’ll just presume you’re a Pythonista
Here’s the folder structure we’re dealing with:
foo-project/
| Dockerfile
| Makefile
| app.py
| requirements.txt
We have a Dockerfile (naturally), along with a Makefile to allow us to more easily build our docker image. We also have an application script + a set of dependencies. Nice and simple.
So here’s what the Dockerfile looks like:
FROM python:3.6.1
WORKDIR /tmp
RUN apt-get update -y
RUN apt-get install -y git ncurses-dev
RUN git clone https://github.com/vim/vim.git && cd vim && ./configure --enable-python3interp=yes && make && make install
ADD ./requirements.txt /app/requirements.txt
RUN pip install -r /app/requirements.txt
COPY .vim /root/.vim
COPY .vimrc /root/.vimrc
WORKDIR /app
As you can see we’re building our Docker image from an official Python base image (at the time of writing it’s the latest Python version).
We jump into a tmp directory so that we can install some dependencies required to get our setup just how we want it. So this means installing git
so we can clone down the latest build of Vim and we also install ncurses-dev
which is necessary in order to compile Vim.
After that we copy our requirements.txt
file into the image and install the packages specified inside that file. We also add in our .vim
directory and .vimrc
file to the image.
Note: the Makefile has a command for copying
.vim/.vimrc
into the current project directory, asdocker build
has a specific context environment that is effectively the location of the Dockerfile
Next we have the Makefile:
copy_vim_files:
@if [ ! -d "./.vim" ]; then cp -r "$$HOME/.vim" ./.vim; fi
@if [ ! -f "./.vimrc" ]; then cp "$$HOME/.vimrc" ./.vimrc; fi
remove_vim_files:
@rm -rf ./.vim
@rm ./.vimrc
build: copy_vim_files
@docker build -t python-container-with-vim .
run: build
@docker run -it -v "$$(pwd)":/app python-container-with-vim /bin/bash
clean: remove_vim_files
-@docker rmi -f python-container-with-vim &> /dev/null || true
rebuild: clean run
There’s lots going on in there, but the important thing to know is that to start up your docker container (with Python and Vim pre-installed) is to use:
make run
This will copy over the host .vim/.vimrc
directory/files, then build a new image and then call docker run ...
where it’ll mount the project directory as a volume into the running container.
Once you’re inside the container just execute vim app.py
and off you go writing code.
Just for completion, here is our application script:
print('hi')
food = "is a thing" # if linting is installed properly this will error
def hello(message):
"""
My summary line starts capitalized and ends with a period.
my bigger description is going here
so pay attention to what it says
"""
print(message)
The above script has some ‘linting’ issues, so if our packages (see below) are installed correctly then we should see Vim highlight the issue to us.
flake8==3.2.1
flake8-deprecated==1.1
flake8-docstrings==1.0.3
flake8-mock==0.3
flake8-quotes==0.9.0
mypy==0.501
pep8-naming==0.4.1
pylint==1.6.4
pytest==3.0.5
We can see from the requirements.txt
file that we’ve installed a few different linters along with the MyPy static analysis tool.
That’s it really. You can reuse the Dockerfile and Makefile for all your projects as they don’t do anything specific to this project. Just setup the docker image/container so you can execute make run
and start developing.
Go
You have a Go project you need to work on.
I won’t explain how Go works,
I’ll just presume you’re a Gopher
Here’s the folder structure we’re dealing with:
bar-project/
| Dockerfile
| Dockerfile-compile
| Godeps
| Makefile
| app.go
So here’s our main Dockerfile:
FROM golang:1.8
RUN apt-get update -y
RUN apt-get install -y wget git ncurses-dev time
WORKDIR /tmp
RUN git clone https://github.com/vim/vim.git && cd vim && make && make install
WORKDIR /go/src
COPY .vim /root/.vim
COPY .vimrc /root/.vimrc
COPY ./Godeps /go/src
RUN wget https://raw.githubusercontent.com/pote/gpm/v1.4.0/bin/gpm && chmod +x gpm && mv gpm /usr/local/bin
RUN gpm install
RUN cp -r ./github.com /github.com # backup packages to root to prevent volume mount removing it
# Install Go binaries that are utilised by the vim-go plugin:
# https://github.com/fatih/vim-go/blob/master/plugin/go.vim#L9
#
# We don't manually install them, we let vim-go handle that
# We use vim's `execute` command to pipe commands
# This helps avoid "Press ENTER or type command to continue"
RUN time vim -c "execute 'silent GoUpdateBinaries' | execute 'quit'"
Again we’re not doing anything too crazy (not until the end, which I’ll explain). We’re building a new image from an official base image, then we’re installing dependencies that allow us to manually compile the Vim editor.
Next we copy over our vim files and the Godeps
dependencies file and we install our dependency manager gpm and install the packages we want to use within our application.
Next we back up the installed depedencies (./github.com
) to another directory. The reason we do that is because when we mount our host project directory into the running container we will end up accidentally replacing the installed packages.
Finally we run vim and pass it a script to be executed once Vim has loaded. What this does is allow the statically built image to include the updated set of depedencies that the vim-go plugin requires. I could have installed them manually, but then using the built in command provided by vim-go means I don’t have to ensure my list of go tools still matches up to what vim-go is using.
The downside to this is that when you build the image, you’ll see (for ~20-30 seconds) Vim started and you wont be able to interact with it at all during that time. This is because it’s installing the dependencies it uses. But after that, Vim will close and you’ll be placed at the containers shell prompt. From there you can run Vim and start coding.
Here’s the Go Makefile (it works similarly to the Python one):
bin := "/usr/local/bin/fastly"
vim_dir := "./.vim"
vimrc := "./.vimrc"
container_env := "go-container-with-vim"
container_compiler := "go-compiler"
copy_vim_files:
@if [ ! -d $(vim_dir) ]; then cp -r "$$HOME/.vim" $(vim_dir); fi
@if [ ! -f $(vimrc) ]; then cp "$$HOME/.vimrc" $(vimrc); fi
remove_vim_files:
@rm -rf $(vim_dir) &> /dev/null || true
@rm $(vimrc) &> /dev/null || true
remove_compiled_files:
@rm ./fastly.{darwin,linux,windows.exe} &> /dev/null || true
clean: remove_vim_files remove_compiled_files
@docker rmi -f $(container_env) &> /dev/null || true
@docker rmi -f $(container_compiler) &> /dev/null || true
uninstall: clean
@rm $(bin) &> /dev/null || true
build: copy_vim_files
@docker build -t $(container_env) .
dev: build remove_vim_files
@docker run -it -v "$$(pwd)":/go/src $(container_env) /bin/bash
rebuild: clean run
compile:
@docker build -t $(container_compiler) -f ./Dockerfile-compile .
@docker run -it -v "$$(pwd)":/go/src $(container_compiler) || true
copy_binary:
cp ./fastly.darwin $(bin)
install: compile copy_binary remove_compiled_files
One thing I did this time was change the make run
for make dev
as I feel that’s more indicative of what we’re doing (the ‘run’ suggests we’re running our application when we’re really just wanting to drop into a development environment).
There’s a few more steps in the Go Makefile and that’s just for the purposes of having a separate Dockerfile for compiling our application. The following is the other Dockerfile we have in our project:
FROM golang:1.8
WORKDIR /go/src
COPY ./Godeps /go/src
COPY ./compile.sh /go/src
RUN apt-get update && apt-get install wget
RUN wget https://raw.githubusercontent.com/pote/gpm/v1.4.0/bin/gpm && chmod +x gpm && mv gpm /usr/local/bin
RUN gpm install
RUN cp -r ./github.com /github.com # backup packages to root to prevent volume mount removing it
CMD ["./compile.sh"]
This Dockerfile does much the same thing: get dependency file, install those specified dependencies, then back them up to another directory.
This time though, when the container is run we use a separate script as the CMD
value as our script was getting quite long (as you’ll see).
Here is the contents of compile.sh
:
Note: make sure you
chmod +x ./compile.sh
from your host
#!/bin/sh
# copy packages back into our source code directory
cp -fr /github.com ./github.com
# compile application for the major operating systems
gox -osarch='linux/amd64' -osarch='darwin/amd64' -osarch='windows/amd64' -output='fastly.{{.OS}}'
# run the relevant compatible compiled binary for this container's OS
./fastly.linux
The tasks we run are:
- copy the backed up dependencies back into the mounted project directory
- build the app using the default compiler for the OS †
- execute the compiled binary to show it can run correctly inside the container
† this means our compiled binary will be a linux based binary, so you can’t run it on your host machine if it’s not linux based (e.g. I’m using macOS). You’ll see that to allow me to compiled my application for multiple OS’s I’ve installed Gox
Now here is our Go application, it simply uses the logging dependency we’ve installed and that’s it. Nothing too fancy necessary for this example.
package main
import (
"fmt"
log "github.com/Sirupsen/logrus"
)
func init() {
log.SetLevel(log.DebugLevel) // otherwise would not be shown
}
func main() {
fmt.Println("Hello World!")
logger := log.WithFields(log.Fields{
"name": "hello-world-app",
})
logger.Debug("this is my debug log message")
logger.Info("this is my info log message")
logger.Warn("this is my warn log message")
logger.Error("this is my error log message")
logger.Fatal("this is my Fatal log message")
logger.Panic("this is my Panic log message") // we don't actually reach here
}
Here is our dependency file’s content, where we can see the Logrus dependency we’ve specified (as well as Gox for the purposes of our container responsible for compiling our application for multiple OS’):
github.com/mitchellh/gox c9740af9c6574448fd48eb30a71f964014c7a837
github.com/sirupsen/logrus 10f801ebc38b33738c9d17d50860f484a0988ff5
Mounting Volumes
Just remember that when making changes inside the container, because you’ve mounted your host project directory as a volume, if you make a change or add a new file or compile something inside of the container; then it’ll be available on your host machine.
This is all fine, but you might want to look at setting up a .gitignore
file to ensure you don’t accidentally commit any unwanted items into your git repository.