Actually, Docker for a dummy. Me. To be honest, I have always been scared of DevOps. Whenever reading a Rails tutorial, I would skip the part about deployment. If it worked on my machine, it was good enough for me and I moved on. Obviously this attitude would not hold up in real life. Real life caught up with me quickly and learning Docker was the first thing I had to do on my new job.
Picture your typical Rails app that allows users to register (data persistence with, let’s say, Postgres) and submit a photo, which is then resized (image manipulation with, let’s say, ImageMagick). Picture new developers that need to start working on this app. You can specify the Ruby version your app uses in
.ruby-version. You can specify the gems and their versions your app depends on in a
Gemfile. But Bundler can’t install everything. You would have to trust your developers to install the specific versions of Postgres and ImageMagick you asked for. They probably will if they haven’t already got these installed, but what about a situation where they are currently working on some other app that uses those as well, but some older versions? The point is, developers will run the app on different operating systems, with different versions of external dependencies (like Postres and ImageMagick) and those discrepancies always lead to the infamous “but it works on my machine!”. Docker allows all developers to run the app in exactly the same environment. What is more, this can be exactly the same environment as the production environment. Because in the end of the day, your web app doesn’t have to run on all the operating systems with all the possible versions of all the dependencies. It just have to run on your web server.
What is Docker?
Docker is a tool that allows you to pretend there are tiny computers inside your computer. Those tiny computers are called containers, and the computer they run on is called a host OS. Containers behave like separate operating systems, each having their own independent share of resources (filesystem, CPU, memory). But they are not really separate, they all share the host OS’s kernel.
The host OS has to be Linux, but that doesn’t mean you can’t use Docker on Mac OS X or Windows. It just means you need another layer of indirection. Docker provides a tool called Docker Machine that allows you to create a VirtualBox virtual machine with Linux to use as the Docker host OS.
Step by step example - a Guestbook Rails app
It will be probably easier to understand the difficult concepts of the Docker world if I present them on a familiar ground. Let’s say we have a very simple Guestbook Rails app. Anyone can leave a short message that gets persisted in a database (we will use Postgres). Normally we would have both Postgres and the Rails app running on our computer. The goal is to run them both in separate Docker containers. I will assume we are working on OS X. Please notice that this post is about setting up a development environment. What that means is that we need a convenient way to change code and add gems on the go, but we don’t have to worry much about security, because the app will only be accessible from our machine.
Step 1 - a bare Rails app
It has a single
GuestbookEntry model with a single
body text field.
GuestbookEntriesController has two actions,
index.html.erb view lists all guestbook entries and provides a form to add a new one.
Step 2 - build a Docker image for our Rails app
Docker images are the basis of containers. An image is like a description of the environment you want to run in the container. You specify which operating system you want to start from, which additional tools and libraries to install, which files from your computer to copy to the image and so on.
Agenda for step 2
Let’s imagine that we have an empty computer. What we need to do before we can setup our Rails app is:
- Install an operating system
- Install Ruby
- Install Bundler
- Copy our app’s source code
But don’t worry! We won’t really have to install an operating system on our own. Every Docker image can (and has to) have a base image. That means that there are images with operating systems like Ubuntu or Debian already installed, waiting for us to use them. They live in the Docker Hub. Check out the most popular Docker images that we can use. The image that interests us is Ruby. I will use
ruby:2.3.0. It is based on Debian Jessie and has Ruby v2.3.0 and Bundler installed.
A file that specifies the steps necessary to build a Docker image is called a
Dockerfile. The format of the
Dockerfile is simple:
1 2 # comment INSTRUCTION arguments
All the possible instructions can be found in the Dockerfile reference.
1. an OS, 2. Ruby & 3. Bundler all in one
Let’s start building our
docker-rails-example/guestbook/Dockerfile step by step.
1 FROM ruby:2.3.0
We specify the base image for our Rails app image.
1 RUN apt-get update && apt-get -y install nodejs
4. The rest
1 COPY . /guestbook
We copy the current directory (the directory the Dockerfile is in), that is
docker-rails-example/guestbook, from our machine to
/guestbook on the Docker container.
There is one little problem. Using the
COPY command we copy everything from
docker-rails-example/guestbook/, including unnecessary (for our app to run) files like
Dockerfile itself. Fortunately, there is a quick solution to ignore some files completely and that is creating a
.dockerignore file. It works just as
1 2 3 4 5 6 # docker-rails-example/guestbook/.dockerignore Dockerfile .dockerignore .gitignore tmp/* log/*
Going back to the
1 WORKDIR /guestbook
We change the working directory to the directory where the source code for our app lives.
1 CMD bin/rails s
We specify the default command that will be executed when we run a container from our image. Whatever string is after
CMD will be used as an argument to
/bin/sh -c. This simply means that Docker will run the Bourne shell telling it to run the command specified after the
-c flag. As a reasonable default, we want to run the Rails server. But that doesn’t mean we won’t be able to do anything else!
Putting all of this together:
1 2 3 4 5 6 7 8 FROM ruby:2.3.0 RUN apt-get update && apt-get -y install nodejs COPY . /guestbook WORKDIR /guestbook CMD bin/rails s
We’re now ready to build and run the image! But be aware, this is not the end of our work. Our app will not start, because it will not connect to the database just yet. Still, let’s learn how to build the image.
Building the image
1 $ docker build -t guestbook guestbook
-t guestbook part means that we will call the image
guestbook, and the second
guestbook is a path to the directory with the
Dockerfile. This command assumes that it is run inside
Let’s list all the images to make sure we have indeed built it:
1 2 3 4 $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE guestbook latest 294781cb78ac 5 seconds ago 725.6 MB ruby 2.3.0 cc0ac307dc6d 2 days ago 725.5 MB
Running a container from the image
And now the glorious moment we have been waiting for! Let’s run our image:
1 2 3 4 5 $ docker run guestbook (...)/bundler/spec_set.rb:94:in `block in materialize': Could not find rake-11.1.1 in any of the sources (Bundler::GemNotFound) # (...) a long stack trace
Well, not so glorious. But this is good, this makes sense. We tried to run our default command
bin/rails s without setting up the app first. Let’s run
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 $ docker run guestbook bin/setup == Installing dependencies == The following gems are missing (...) Fetching gem metadata from https://rubygems.org/........... (...) Bundled gems are installed into /usr/local/bundle. (...) == Preparing database == Running via Spring preloader in process 18820 could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"? (...)
Exactly what we have been excepting.
bin/rake db:setup fails, because there is no Postgres running on our container.
There is another major problem here. Notice that Bundler installed the gems to
/usr/local/bundle. That of course is a directory on the container. But the container stops whenever the
bin/setup command is done. If we run another container from the guestbook image, it won’t have anything inside
/usr/local/bundle and we will have to install the gems all over again before doing anything else! What is more, currently we are copying our code when building the image. That means that if we want to change anything in the code, we would have to rebuilt the image, run a container from that image doing the setup all over again. This is awful. We will fix that in step 3 before moving on to running Progres.
Step 3 - make our lifes easier with volumes
Containers run from the same image do not share filesystems. Whatever was copied to the image in the
Dockerfile will be accessible to all containers run from that image, but if a container creates files, they will not be added neither to the image nor to other containers run from the same image.
It would be awesome if we could keep the code and the gems on our machine (OS X) and have the containers read from that. There is a way to achieve this. It’s called data volumes. Well, data volumes allow us to mount a directory from the host OS (Linux VM in our case) on the container. The part where we mount a directory from OS X on the host OS is done by VirtualBox via shared folders. When Docker Machine created the default Linux host machine, it added
/Users from OS X as a shared folder.
Whenever you’re in doubt about what’s actually happening on the Linux VM, you can always open it via VirtualBox and talk to it.
Agenda for step 3
Back to our Rails app. What we will do is:
- Do not copy the source code while building the image
- Tell Bundler not to install gems in
/usr/local/bundle, but rather somewhere that will be shared between OS X and the container, like
- Mount the directory with the source code
1. Do not copy & 2. Set environment variables for Bundler
1 2 3 4 5 6 7 8 9 10 11 12 13 14 FROM ruby:2.3.0 RUN apt-get update && apt-get -y install nodejs ENV GEM_HOME /guestbook/.bundle ENV BUNDLE_PATH="$GEM_HOME" \ BUNDLE_BIN="$GEM_HOME/bin" \ BUNDLE_APP_CONFIG="$GEM_HOME" ENV PATH $BUNDLE_BIN:$PATH COPY . /guestbook WORKDIR /guestbook CMD bin/rails s
Now we have to build the image again:
1 $ docker build -t guestbook guestbook
3. Run a container with mounted volumes
1 $ docker run -v `pwd`/guestbook:/guestbook guestbook bin/setup
-v `pwd`/guestbook:/guestbook part tells Docker to mount
docker-rails-example/guestbook from the host OS at
/guestbook on the container, and as we have established before,
docker-rails-example/guestbook on the host OS is the same as
docker-rails-example/guestbook on OS X via VirtualBox shared folder.
bin/setup failed again because it still can’t connect to Postgres. We will take care of that in the next step. Try running this command again. You will see that this time Bundler is happy, because Gemfile’s dependencies are satisfied. All the gems are now on OS X inside
Step 4 - linking our app to Postgres
We want to run another container, with Postgres. And we want our current Rails app container to know about it.
Agenda for step 4
- Run two containers, one with Postgres, another with the Rails app
- The container with the app has to know about the container with Postgres and it needs to be able to connect to it
1. Composing containers
It’s time we start composing our containers. We need to create a
docker-compose.yml file. It will define which containers to run and how they are linked together. It will then allow us to run all the containers with one command.
1 2 3 4 5 6 # docker-rails-example/docker-compose.yml version: "2" services: postgres: image: postgres:9.5
Firstly, we specify the version of the compose file format, version 2 is recommended. Secondly, we define a container called
postgres to be run from the image
postgres:9.5 that will be downloaded from the Docker Hub. That’s all. Seriously, it’s that easy to run Postgres. Let run the container. When composing containers, we will be using a slightly different command to build and run everything. Since Postgres is an already built image, we just need to run it:
1 2 3 $ docker-compose up -d Creating network "dockerrailsexample_default" with the default driver Creating dockerrailsexample_postgres_1
-d flag means that the process will be detached. If we didn’t include it, we would be getting logs in the terminal.
Ok, how do we know anything is happening? We can list all containers currently running:
1 2 3 $ docker ps (...) IMAGE (...) STATUS PORTS (...) (...) postgres:9.5 (...) Up About a minute 5432/tcp (...)
It’s definitely running. Let’s stop it with
docker-compose down and add the Rails app.
2. Linking containers
Adding the Guestbook app to
1 2 3 4 5 6 7 8 9 10 11 12 13 14 version: "2" services: postgres: image: postgres:9.5 guestbook: build: ./guestbook volumes: - ./guestbook:/guestbook links: - postgres ports: - 3000:3000
Let’s break it down:
docker-composewhere to look for the
Dockerfilefor this container
volumesis a list of data volumes to be mounted on this container, it works just as the
linksis a list of services that will be know to the
portsis a list of exposed ports. By adding
3000:3000we specify that port
3000on the host OS (the first one) is mapped to port
3000on the container (the second one)
Rails database config
But that’s not enough. Notice that Rails reads the database configuration form
config/database.yml. We need to change it as well.
1 2 3 4 5 6 7 default: &default adapter: postgresql encoding: unicode pool: 5 username: postgres password: host: postgres
To find out the username and the password, we had to look at the Postgres image’s docs. They can be easily found in the Docker Hub. The user is specified by the environment variable
POSTGRES_USER, and the password by
POSTGRES_PASSWORD. It defaults to user
postgres with no password. Notice that
host has to match the name of the service we defined as a link for our app.
Rails server tweak
One more tweak to the
1 CMD rm -f tmp/pids/* && bin/rails s -p 3000 -b '0.0.0.0'
We have to bind Rails server to ip
0.0.0.0. and make sure that it always runs on port
3000, because that’s the port we exposed. Unfortunately, we also have to remove the file with Rails server’s process id, because for some reason it can’t do it properly by itself when run on a Docker container.
Run all composed containers together
We changed the
Dockerfile for the Rails app, so let’s rebuild first.
1 $ docker-compose build
Let’s run everything with
docker-compose up -d. We have to wait a while for the server to start, but when it does, we can visit
http://192.168.99.100:3000/ in the web browser. We will see a pretty
FATAL: database "guestbook_development" does not exist error.
Let’s set up the database by running
docker-compose run guestbook bin/setup. Please notice that we are using the tool
docker-compose instead of
docker now, even to run single containers.
http://192.168.99.100:3000/ again. It works!
But does it really persist? Let’s add an entry, stop everything and try again.
1 2 3 $ docker-compose down $ docker-machine restart $ docker-compose up -d
FATAL: database "guestbook_development" does not exist again?! That means all the databases are gone! That makes sense. Everything that was created by Postgres in the container disappeared with the container. We will fix it in the next step.
Step 5 - REALLY persist data with Postgres
The problem is: Postgres run in the container saves its data in that container’s filesystem. When the container is stopped, this data can’t be accessed again. When a new container with Postgres is run, it has no recollection of the previous Postgres container’s files.
The solution is: trick Postgres into saving its data to the host OS.
Agenda for step 5
- Find out to which directory Postgres from the
postgres:9.5image saves its data.
- Mount a named volume at that directory
We need to consult the docs again. It’s easy to find that Postgres stores its data where the
PGDATA environment variable points to, the default value is
2. Named volumes
Whenever we mount a directory from the host OS to a Docker volume, a new directory inside
/var/lib/docker/volumes on the host OS gets created. It has a random name. We can create a volume that has a name, hence it will always be in the same directory inside
/var/lib/docker/volumes, with the same data.
You might wonder why we won’t simply mount
/var/lib/postgresql/data from the host OS. It’s because whenever the Docker Machine is restarted, it resets its filesystem (at least on OS X). As far as we’re concerned, everything outside
/var/lib/docker/ is not persisted. That’s why a named volume will survive a Docker Machine restart.
Let’s tweak our
docker-compose.yml to look like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 version: "2" volumes: postgres_data: services: postgres: image: postgres:9.5 volumes: - postgres_data:/var/lib/postgresql/data guestbook: build: ./guestbook volumes: - ./guestbook:/guestbook links: - postgres ports: - 3000:3000
All that is left now is to run the containers again, setup the app, check that it works, restart everything and check again.
1 2 3 4 5 6 7 $ docker-compose up -d $ docker-compose run guestbook bin/setup # Check 192.168.99.100:3000 in the web browser $ docker-compose down $ docker-machine rest $ docker-compose up -d # Check 192.168.99.100:3000 in the web browser again
Everything should be fine now. Change something in the code and refresh the browser. It works! You can run all the typical commands like so
docker-compose run guestbook bin/rake db:seed. However, in the rare occasion of adding a new gem, you will have to restart the containers.
Read containers logs
1 $ docker-compose logs
Or even better, use the GUI app - Kitematic.
Read the build log
Not everything will show up in the terminal when building. Redirect the output of the build command to a file to get a more detailed log:
1 $ docker-compose build > buildlog.txt
Clean up the mess
Running all these containers left quite a mess behind. Every time we have run a container, it didn’t remove itself after stopping.
List all containers (both running and stopped)
1 $ docker ps -a
Remove all containers (both running and stopped)
1 $ docker rm $(docker ps -aq)
List all the images
1 $ docker images
Remove all images
1 $ docker rmi $(docker images -q)
Run everything with the
1 $ docker-compose run --rm bin/rake db:seed
Now when the container stops, it will remove itself.
Where did the
192.168.99.100 come from?
It’s the IP of the Linux VM. You can check it by running
If you don’t want to type the IP address every time, you can map a nice hostname to it:
1 2 # /etc/hosts 192.168.99.100 dockermachine