Elixir has seen fast adoption in Wego with multiple services already in production after the weekend project UltronEx became a daily driver for monitoring real time data it has spurred a movement to explore new options and build robust resilient systems.
As We run almost all our work loads in containers we take advantage of the same build pipelines for building container images to run CI/CD as well. Elixir runs on BEAM and has some nice features such as hot redeploys that is something anti to the concept of containers where they are immutable images none the less it does not mean Elixir is not container friendly. Mix, the elixir build tool similar to what Rake is for Ruby and has great support to generate a bundle with ERTS called a Release that you can take and run anywhere as long as it is the same platform it was built on. This is what works really well with containers keeping the image size significantly small without any external dependencies. Even though this makes the Elixir Release easy to work with it also strips it of its tool chain that you require to perform actions such as running migration or executing tasks as there is no Mix available in a Release. The below chart shows the different approaches and their features
One might ask why use a Release then for that there is a great thread on Elixir forum on why to Always use Releases. Let’s say If you were to package your app with Mix and run it using mix run --no-halt
to overcome issues with missing Mix what difference does it make on the container size ? The elixir image for 1.11.2-erlang-23.1.3
is about 214MB
which is used as the base to either build a release in a multi stage Dockerfile or use it to run the app using Mix. Compared to that Debian
image is only 69.2MB
. When we use an Elixir Release for a Phoenix app and copy that to a stage based on Debian the final image size comes to 101MB
whereas if we were to use Mix and not generate a release the final image size is 627MB
a whopping 6X the release image size.
That is some significant advantage releases have over other approaches and also goes on to show the impact multi stage build files can have over image sizes.
So that sorts that we should always use Releases but how to then get the lost functionality of Mix ? For this Elixir and Docker both do their part to seamlessly make things similar to what you could do with a Ruby CI/CD containerised process with rake. For running database migration with releases Elixir has a work around with Ecto Migrator where in you create a separate file that executes the migrations, it is a standardised code and should actually IMO be part of Ecto
defmodule MyApp.Release do
@app :my_appdef migrate do
load_app()for repo <- repos() do
{:ok, _, _} = Ecto.Migrator.with_repo(repo, &Ecto.Migrator.run(&1, :up, all: true))
end
enddef rollback(repo, version) do
load_app()
{:ok, _, _} = Ecto.Migrator.with_repo(repo, &Ecto.Migrator.run(&1, :down, to: version))
enddefp repos do
Application.fetch_env!(@app, :ecto_repos)
enddefp load_app do
Application.load(@app)
end
end
After creating your release you can run you migrations in production with eval
for e.g
$ _build/prod/rel/my_app/bin/my_app eval "MyApp.Release.migrate"
Migration is a special case and Ecto has some nice features to access the Database to make this all possible but for running some tasks in production environment requires a separate approach and some understanding of an Elixir application or for a web application built in Phoenix. Every Elixir application is built of multiple applications where a Phoenix app is a list of applications where Web and Database are separate applications which also gives Elixir applications its robustness where any single application can not bring down your entire Service. For this you can take lead from the elixir release docs and base understanding of elixir application to formulate your application container image to be used to execute tasks and run the application service as well. Let us take a Phoenix app GhostRider
as an example. Here we split the application children into two separate lists base
and web
. Based on what is required we can handle what children load from an environment variable which can conveniently be passed to a container at runtime.
defmodule GhostRider.Application do
# See https://hexdocs.pm/elixir/Application.html
# for more information on OTP Applications
@moduledoc falseuse Applicationdef start(_type, _args) do
base = [
# Start the Ecto repository
GhostRider.Repo,
{Finch, name: GhostRiderFinch}
]web = [
# Start the Telemetry supervisor
GhostRiderWeb.Telemetry,
# Start the PubSub system
{Phoenix.PubSub, name: GhostRider.PubSub},
# Start the Endpoint (http/https)
GhostRiderWeb.Endpoint
# Start a worker by calling: GhostRider.Worker.start_link(arg)
# {GhostRider.Worker, arg},
]children =
case System.get_env("APP_LEVEL") do
"TASK" -> base
_ -> base ++ web
end# See https://hexdocs.pm/elixir/Supervisor.html
# for other strategies and supported options
opts = [strategy: :one_for_one, name: GhostRider.Supervisor]
Supervisor.start_link(children, opts)
end# Tell Phoenix to update the endpoint configuration
# whenever the application is updated.
def config_change(changed, _new, removed) do
GhostRiderWeb.Endpoint.config_change(changed, removed)
:ok
end
end
Though this approach would work but applications only start when a release is started. To overcome this we need to use the eval
to execute a wrapper function bound to perform operation but in order to execute that operation we need to start our application using Application.ensure_all_started(@app)
along with the environment variable controlling which children should be loaded.
With Migrations and Task sorted and capable of working with containers the last piece of the CI/CD puzzle is how do you run your test when there is no Mix in your release and the release is byte code and no test folder is available ? This is where docker shines with its multi stage builds and we leverage docker-compose to target a stage to be able to execute tests in the build to ensure they are consistent across different technology stacks we have.
Now some docker
and docker-compose
recipes to bring this all together to work in a CI/CD pipeline. One thing I have to admit is lack of resources around containerising Elixir/Phoenix applications, something the community should work towards improving. Below is a file for building a Phoenix application and you can see there is a stage called tester
this is what we will target using docker-compose
# https://elixirforum.com/t/could-use-some-feedback-on-this-multistage-dockerfile-1st-elixir-phoenix-deployment/30862/10?########################
### Dependency stage ###
########################
FROM hexpm/elixir:1.11.2-erlang-23.1.3-debian-buster-20201012 AS base# install build dependencies
RUN apt-get -qq update && \
apt-get -qq -y install build-essential npm git python --fix-missing --no-install-recommends# prepare build dir
WORKDIR /appARG MIX_ENV
ARG RELEASE_ENVENV LANG=C.UTF-8 LC_ALL=C.UTF-8# Update timezone
ENV TZ=Asia/Singapore# install hex + rebar
RUN mix local.hex --force && \
mix local.rebar --force# set build ENV
ENV MIX_ENV=${MIX_ENV}
ENV RELEASE_ENV=${RELEASE_ENV}COPY mix.exs mix.lock ./
COPY config config# install mix dependencies
RUN mix deps.get --only ${MIX_ENV}
RUN mix deps.compileCOPY lib ./lib
COPY priv ./priv
#########################
####### Test Stage ######
#########################FROM base as tester
WORKDIR /app
COPY test test########################
# Build Phoenix assets #
########################
# Using stretch for now because it includes Python
# Otherwise you get errors, could use a smaller image though
FROM node:14.15.3-stretch AS assets
WORKDIR /appCOPY --from=base /app/deps /app/deps/
COPY assets/package.json assets/package-lock.json ./assets/
RUN npm --prefix ./assets ci --progress=false --no-audit --loglevel=errorCOPY lib ./lib
COPY priv ./priv
COPY assets/ ./assets/RUN npm run --prefix ./assets deploy#########################
# Create Phoenix digest #
#########################
FROM base AS digest
WORKDIR /app# set build ENV
ENV MIX_ENV=${MIX_ENV}
ENV RELEASE_ENV=${RELEASE_ENV}COPY --from=assets /app/priv ./priv
RUN mix phx.digest#######################
#### Create release ###
#######################
FROM digest AS release
WORKDIR /appARG MIX_ENV
ARG RELEASE_ENV
ENV MIX_ENV=${MIX_ENV}
ENV RELEASE_ENV=${RELEASE_ENV}COPY --from=digest /app/priv/static ./priv/staticRUN mix do compile, release#################################################
# Create the actual image that will be deployed #
#################################################
FROM debian:buster-slim AS deploy# Install stable dependencies that don't change often
RUN apt-get update && \
apt-get install -y --no-install-recommends \
apt-utils \
openssl \
curl \
wget && \
rm -rf /var/lib/apt/lists/*# Set WORKDIR after setting user to nobody so it automatically gets the right permissions
# When the app starts it will need to be able to create a tmp directory in /app
WORKDIR /appARG MIX_ENV
ARG RELEASE_ENVENV MIX_ENV=${MIX_ENV}
ENV RELEASE_ENV=${RELEASE_ENV}ENV LANG=C.UTF-8 LC_ALL=C.UTF-8# Update timezone
ENV TZ=Asia/SingaporeHEALTHCHECK --start-period=10s \
--interval=15s \
--timeout=5s \
--retries=3 \
CMD curl -sSf http://localhost:8080/heartbeat || exit 1COPY --from=release /app/_build/${MIX_ENV}/rel/ghost_rider ./ENV HOME=/app
# Exposes port to the host machine
EXPOSE 8080CMD ["bin/ghost_rider", "start"]
A standard docker-compose
file to run the application would be similar to this which would spin up a Postgres database and start the application for you.
version: "3.7"
services:
ghost-rider:
build:
context: .
args:
MIX_ENV: "${MIX_ENV}"
RELEASE_ENV: "${RELEASE_ENV}"
image: wego/ghost-rider
depends_on:
- ghost-rider-db
restart: on-failure
ports:
- "8080:8080"
environment:
MIX_ENV: "${MIX_ENV}"
RELEASE_ENV: "${RELEASE_ENV}"
PORT: "8080"
SECRET_KEY_BASE: "${SECRET_KEY_BASE}"
DB_NAME: ghost-rider
DB_PASSWORD: ghost-rider
DB_HOST: ghost-rider-db
APP_LEVEL: WEBghost-rider-db:
image: postgres
restart: always
ports:
- 5432:5432
environment:
POSTGRES_PASSWORD: ghost-rider
POSTGRES_DB: "${POSTGRES_DB}"
PGDATA: /var/lib/postgresql/data/pgdata
volumes:
- "postgresql-data:/var/lib/postgresql/data"volumes:
postgresql-data:
To build and run your container image you would execute
$ docker-compose -f docker-compose.yml up -d
To run your migration
$ docker-compose run --rm ghost-rider bin/ghost_rider eval "Release.migrate"
To run the test you would need to create another docker-compose
file to override the default values and define the target it should execute up to. If you want you could create a completely new file as well
version: "3.7"
services:
ghost-rider:
build:
target: tester
args:
- MIX_ENV=test
- RELEASE_ENV=test
image: wego/ghost-rider-test
ports:
- "8080:8080"
environment:
- MIX_ENV=test
- RELEASE_ENV=test
- DB_NAME=ghost-rider-test
- DB_PASSWORD=ghost-rider
- DB_HOST=ghost-rider-db
With this you can setup your test database and run test simply by doing
$ docker-compose -f docker-compose.yml -f docker-compose.test.yml run --rm ghost-rider mix ecto.setup$ docker-compose -f docker-compose.yml -f docker-compose.test.yml run --rm ghost-rider mix test
All it takes is some understanding of Elixir and it can be used with Docker tools as any other language to build images and perform CI/CD without any friction.
I hope this helps with containerising Elixir and Phoenix applications as they are amazing technologies along with Docker and containers that make things really simple build pipelines for CI/CD.
Originally published at https://geeks.wego.com on December 24, 2020.