Deploying production, staging and review apps environments automatically with Kamal on CI/CD
Guillaume Briday
9 minutes
One of the features I truly appreciate about PaaS platforms like Heroku, Netlify, or Vercel is review apps. The idea is simple but powerful: spin up a temporary environment to test new functionality from a Git branch before pushing it to production. It's such a smooth way to catch issues early and share progress with your team.
I wanted to bring that same experience to my deployments with Kamal.
If you’re new to Kamal, I’ve written quite a few blog posts on it that I suggest you check out. They’ll give you a solid foundation for understanding the setup.
ℹ️ This blog post is part of a series about Kamal. Many of the configurations and concepts discussed here remain consistent with those covered in the previous post, How to deploy Rails with Kamal, PostgreSQL, Sidekiq, and Backups on a single host.
This time around, we’ll be diving into Kamal 2, instead of Kamal 1, exploring its new features and improvements. Let's dive in!
I made a video highlighting the steps featured in this article!
1. Set up your server
Ensure the server has sufficient power, as it will host multiple Rails applications, including at least one production app, one staging app, and several review apps.
2. Configure DNS
Next, configure your DNS settings. Since the subdomains will correspond to branch names, we'll use wildcard subdomains.
To simplify server management, it's helpful to create a CNAME record that acts as an alias for your server. This approach allows you to avoid relying on raw IP addresses, which can be hard to remember, and instead use a friendly alias in tools like Kamal or other configurations.
scw-everest 86400 IN A 192.168.0.1 ; Alias for the server.
timecop 86400 IN CNAME scw-everest.example.com. ; CNAME for production.
*.timecop 86400 IN CNAME timecop.example.com. ; Wildcard for staging and all review apps.
3. Prepare your server
With Kamal, everything runs in Docker, so theoretically, you don't need to install anything additional on your server. Kamal will even install Docker for you automatically.
However, if you've just set up a new server, it's a good idea to configure basic security settings and optimize the default configuration.
To simplify this process, I've created an Ansible playbook to automate these actions. This is especially useful when deploying your application to multiple servers.
If you're interested, you can follow the instructions here:
👉 Kamal Ansible Manager
This playbook handles tasks such as upgrading existing packages and installing essential tools like the UFW firewall, Fail2ban, and NTP.
Once cloned, you can use the CNAME you just created to prepare your server by specifying it in your hosts.ini
file.
[webservers]
scw-everest.example.com ansible_become_method=su ansible_user=root
$ ansible-playbook -i hosts.ini playbook.yml
Finally, reboot
the server, and voilà! Your server is now secure and ready to host your applications.
4. Set up Kamal configuration files
This is the key step where the core setup takes place. We'll rely heavily on Kamal's destination configuration.
The production environment will serve as the default configuration. Below is a simplified example of my config/deploy.yml
file. For this setup, I'm using the server's CNAME alias for the hosts and limiting memory usage to 256 MB.
Additionally, it's necessary to configure the app_port
on the proxy, as the default is not set to 3000.
# Name of your application. Used to uniquely configure containers.
service: my_awesome_app
# Name of the container image.
image: your-name/my_awesome_app
# Deploy to these servers.
servers:
web:
hosts:
- scw-everest.example.com
options:
memory: 256m
job:
hosts:
- scw-everest.example.com
cmd: bundle exec sidekiq
options:
memory: 256m
proxy:
ssl: true
app_port: 3000
host: timecop.example.com
# Inject ENV variables into containers (secrets come from .kamal/secrets).
env:
clear:
RAILS_ENV: production
POSTGRES_USER: 'timecop'
POSTGRES_DB: 'timecop_production'
POSTGRES_HOST: 'timecop-db' # With the pattern: <service_name>-<accessory_name>
secret:
- RAILS_MASTER_KEY
- POSTGRES_PASSWORD
accessories:
db:
image: postgres:16
host: scw-everest.example.com
env:
clear:
POSTGRES_USER: 'timecop'
POSTGRES_DB: 'timecop_production' # The database will be created automatically on first boot.
secret:
- POSTGRES_PASSWORD
directories:
- data:/var/lib/postgresql/data
Next, we'll create a destination for the staging
environment. This environment plays a crucial role, as all the review apps will share the same database with it. Managing a single PostgreSQL instance for both staging and review apps is far more efficient than setting up separate instances for each review app, especially when those instances have no data.
To achieve this, create a file named config/deploy.staging.yml
. The configurations in this file will automatically merge with those in config/deploy.yml
, so you only need to include the settings you want to override.
# Name of your application. Used to uniquely configure containers.
service: timecop-staging
# Name of the container image.
image: timecop-staging
proxy:
host: staging.timecop.example.com
env:
clear:
RAILS_ENV: staging
POSTGRES_USER: 'timecop'
POSTGRES_DB: 'timecop_staging'
POSTGRES_HOST: 'timecop-staging-db' # With the pattern: <service_name>-<accessory_name>
accessories:
db:
image: postgres:16
host: scw-everest.example.com
env:
clear:
POSTGRES_USER: 'timecop'
POSTGRES_DB: 'timecop_staging' # The database will be created automatically on first boot.
secret:
- POSTGRES_PASSWORD
directories:
- data:/var/lib/postgresql/data
By default, secrets in Kamal are scoped to specific destinations. However, to simplify management, we can share secrets across multiple destinations. To do this, place all shared secrets in the .kamal/secrets-common
file.
You can still define destination-specific secrets in .kamal/secrets
for production or .kamal/secrets-staging
for staging if needed.
Before the first deployment, be sure to run the kamal setup
command to initialize the configuration.
$ kamal setup # For the production
$ kamal setup -d staging # For the staging
Both applications should now be accessible at timecop.example.com
and staging.timecop.example.com
. 🚀
5. Configuring the review-app destination
This is where the real magic happens.
Since Kamal is written in Ruby, we can use ERB tags within YAML files to dynamically access environment variables. This technique allows us to deploy review apps dynamically by leveraging environment variables.
The concept is simple: create a new Kamal destination configuration file, such as config/deploy.review-app.yml
.
Instead of hardcoding values in the configuration, you can use ERB tags using the following syntax:
key: <%= ENV['YOUR_VARIABLE'] %>
You can even define default values to use if the environment variable does not exist, like this:
key: <%= ENV.fetch('YOUR_VARIABLE', 'default_value') %>
With that in mind, here's an example of my config/deploy.review-app.yml
destination configuration file:
service: <%= ENV['SERVICE_NAME'] %> # We are going to use the name of the git branch dynamically here, because the name must be unique
image: <%= ENV['SERVICE_NAME'] %>
proxy:
host: <%= ENV['HOST'] %> # We are going to use the name of the git branch dynamically here
env:
clear:
RAILS_SERVE_STATIC_FILES: true
RAILS_ENV: staging
POSTGRES_USER: 'timecop'
POSTGRES_DB: 'timecop_staging'
POSTGRES_HOST: 'timecop-staging-db' # With the pattern: <service_name>-<accessory_name>
HOST: <%= ENV['HOST'] %>
As shown in the example, I'm using the staging database for the review apps. You could take a similar approach for other resources like Redis, S3 buckets, and more. The idea here is to avoid the inconvenience of having an empty database on a newly deployed review app. I also prefer this approach because it avoids the need to create seed data repeatedly to populate the database, this choice is entirely up to you though.
That said, if you prefer, you can dynamically provision a separate database for each review app, just as we've done for staging
and production
. It's a matter of finding the balance between simplicity and the degree of isolation you require.
You can test this configuration directly in the command line before integrating it into our CI/CD pipeline:
SERVICE_NAME=new-feature HOST=new-feature.timecop.guillaumebriday.fr kamal deploy -d review-app
To tear down the review app, simply ensure that the variables remain consistent:
SERVICE_NAME=new-feature HOST=new-feature.timecop.guillaumebriday.fr kamal remove -y -d review-app
And it should work as expected! 🚀
6. Setting up GitLab CI
Let's take the next step: configuring our pipeline to automatically deploy a review app whenever a Merge Request is opened on GitLab.
For this example, I'm using GitLab since it's my go-to CI/CD platform. However, the core principles are easily adaptable to other tools like CircleCI, GitHub Actions, or similar platforms. Feel free to adapt the concept to suit your preferred tools and workflow.
For those who need a reference, see: https://docs.gitlab.com/ee/ci/.
Below is a sample .gitlab-ci.yml
file designed for a Ruby on Rails application, pretty standard for this kind of project:
image: circleci/ruby:3.3.6
stages:
- lint
- test
- review
- deploy
include:
- local: '.gitlab/ci/lint.yml'
- local: '.gitlab/ci/test.yml'
- local: '.gitlab/ci/review-apps.yml'
- local: '.gitlab/ci/deploy.yml'
I've divided my configuration into multiple files for better organization, but that's not the focus here. The file we're interested in is the .gitlab/ci/review-apps.yml
. This is where the logic happens for setting up review apps:
.setup_deploy_env: &setup_deploy_env
only:
- merge_requests
image: docker:dind
services:
- docker:dind
variables:
SERVICE_NAME: timecop-${CI_COMMIT_REF_SLUG}
REVIEW_APP_HOST: scw-everest.guillaumebriday.fr
HOST: $CI_COMMIT_REF_SLUG.timecop.guillaumebriday.fr
before_script:
- 'command -v ssh-agent >/dev/null || (apk update && apk add openssh-client)'
- eval $(ssh-agent -s)
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- echo "${REVIEW_APP_HOST}" >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
- apk update && apk add bash curl git yaml-dev build-base openssl-dev readline-dev zlib-dev libffi-dev
- git clone https://github.com/rbenv/rbenv.git ~/.rbenv
- echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bashrc
- echo 'eval "$(rbenv init -)"' >> ~/.bashrc
- source ~/.bashrc
- git clone https://github.com/rbenv/ruby-build.git ~/.rbenv/plugins/ruby-build
- rbenv install 3.3.6 && rbenv global 3.3.6
- gem install kamal
deploy_review_app:
<<: *setup_deploy_env
stage: review
script:
- kamal deploy -d review-app
environment:
name: review/$CI_COMMIT_REF_NAME
url: 'https://${HOST}'
auto_stop_in: 1 day
on_stop: stop_review_app
stop_review_app:
<<: *setup_deploy_env
stage: review
script:
- kamal remove -y -d review-app
when: manual
environment:
name: review/$CI_COMMIT_REF_NAME
action: stop
At first glance, the configuration might seem a bit complex, but once you break it down and consider the steps we've already covered, it becomes quite straightforward and easy to follow.
Here's the key section to focus on:
.setup_deploy_env: &setup_deploy_env
only:
- merge_requests # We want to run this job only when we open a merge request.
image: docker:dind # We use this image to be able to use Docker in Docker (dind). This is required to run kamal deploy and build the image.
services:
- docker:dind
variables: # These environment variables will be used in our `config/deploy.review-app.yml` destination configuration file.
SERVICE_NAME: timecop-$CI_COMMIT_REF_SLUG
REVIEW_APP_HOST: scw-everest.guillaumebriday.fr
HOST: $CI_COMMIT_REF_SLUG.timecop.guillaumebriday.fr
The $CI_COMMIT_REF_SLUG
variable is a predefined environment variable in GitLab CI, automatically provided for your pipelines. For details, see: https://docs.gitlab.com/ee/ci/variables/predefined_variables.html#predefined-variables.
This variable is incredibly useful because it represents a parameterized version of the current branch name. For instance, if you're working on the branch feature/new-homepage
, $CI_COMMIT_REF_SLUG
will automatically be transformed into feature-new-homepage
. This is particularly handy when deploying review apps or managing branch-specific tasks.
In the before_script
section of the .gitlab-ci.yml
file, you'll need to handle the SSH setup and install Ruby to prepare the environment.
Before you can deploy, it's crucial to configure SSH access between GitLab and your server. To do this, use the SSH_PRIVATE_KEY
variable in GitLab CI. This requires adding an SSH key to your server and copying the private key into the GitLab CI/CD variables. For more information, check out the documentation: https://docs.gitlab.com/ee/ci/variables/index.html.
7. Automating deployment with GitLab CI
At this stage, the main job in the pipeline is designed to handle deployment automatically. It simply executes the kamal deploy
command, ensuring that the correct flags and environment variables are passed along.
deploy_review_app:
<<: *setup_deploy_env
stage: review
script:
- kamal deploy -d review-app
environment:
name: review/$CI_COMMIT_REF_NAME
url: "https://${HOST}"
auto_stop_in: 1 day
on_stop: stop_review_app
In this step, the environment
key is utilized specifically for GitLab's environments feature. This allows us to map our review apps to GitLab's environment management tools seamlessly.
Now, whenever you open a Merge Request on a new branch or push new commits, your review app will be deployed automatically! 🚀
What's even better is that you can manage deployments directly from the GitLab UI. You can trigger a new deployment or stop an active one with just a few clicks! An incredibly convenient feature when working with non-tech teams, like QA.
8. Conclusion
And that's a wrap! While there's room for further optimization, such as creating a Docker image in the before_script
to speed up deployments, it ultimately depends on your specific needs and preferences.
It's worth noting that deployments in CI/CD pipelines can sometimes be slow, and there are certainly ways to enhance performance if needed. That said, this approach has worked reliably for me, providing a balance between simplicity and functionality.
Feel free to adapt and optimize it to fit your workflow.
Happy deploying! 🚀