DevOps

How to deploy Rails with Kamal, PostgreSQL, Sidekiq and Backups on a single host

Profil Picture

Guillaume Briday

7 minutes

In the previous post, How to deploy Rails with Kamal and SSL certificate on any VPS, we saw how to deploy a simple Rails application with Kamal and Docker.

But it was intentionally very simple and even if it might be enough for some applications like https://github.com/guillaumebriday/modern-datatables, it does not really reflect how you want to deploy your application in real life.

In real life, you (probably) want to use:

  • PostgreSQL instead of SQLite.
  • Backup (and restore) your database.
  • Use background jobs with Sidekiq (or another queue system).
  • Use Redis for cache.

In this example, we use Ruby on Rails, but the process would be similar with Laravel, Django or any other framework, adapt according to your needs.

So, let's see how to do it!

ℹ️ Most of the configuration and concept, will remain the same compared to the previous post, you should read it before this one.

1. Prepare the server

If you have new servers, you should take a look at my ansible playbook guillaumebriday/kamal-ansible-manager that will configure them properly for Kamal.

Unlike in the previous post, this time, we need to set up accessories and jobs.

Because we are running all of them on the same host, in order to allow them to communicate, we need to create a Docker Network.

More information here (in 🇫🇷): Docker : Les réseaux personnalisés.

Before starting, run this command, on your server:

$ docker network create --driver bridge private

I use the name private but it can be anything, just stay consistent in your configuration.

2. Update the configuration files

The configuration is very similar to the previous one, but we need to adjust few things.

Servers

Let's configure our servers and don't forget to configure our new private docker network:

servers:
  web:
    hosts:
      - 192.168.0.1
    labels:
      traefik.http.routers.my_awesome_app.entrypoints: websecure
      traefik.http.routers.my_awesome_app.rule: Host(`my_awesome_app.com`)
      traefik.http.routers.my_awesome_app.tls.certresolver: letsencrypt
    options:
      network: "private" # This is important!

  job:
    hosts:
      - 192.168.0.1
    cmd: bundle exec sidekiq -q default -q mailers
    options:
      network: "private" # This is important!

Accessories

Now let's configure our accessories with correct directories so the file are saved on the disk and won't be removed between deployments.

accessories:
  db:
    image: postgres:16
    host: 192.168.0.1
    # port: 5432 # Remove this line!
    env:
      clear:
        POSTGRES_USER: "my_awesome_app"
        POSTGRES_DB: "my_awesome_app_production" # The database will be created automatically on first boot.
      secret:
        - POSTGRES_PASSWORD
    directories:
      - data:/var/lib/postgresql/data
    options:
      network: "private" # This is important!

  redis:
    image: redis:7.0
    host: 192.168.0.1
    # port: 6379 # Remove this line!
    directories:
      - data:/data
    options:
      network: "private" # This is important!

You don't need to expose your accessories ports, remove the port key in their configuration, so they won't be accessible from outside Docker for nothing.

Because we use private Docker Network, we can use the name of the containers instead of exposing ports or using external IP address:

env:
  clear:
    RAILS_SERVE_STATIC_FILES: true
    POSTGRES_USER: "my_awesome_app"
    POSTGRES_DB: "my_awesome_app_production"
    POSTGRES_HOST: "my_awesome_app-db" # With the pattern: <service_name>-<accessory_name>
    REDIS_URL: "redis://my_awesome_app-redis:6379/0" # We need to override the REDIS_URL (used by Sidekiq and your cache_store), same pattern: <service_name>-<accessory_name>
  secret:
    - RAILS_MASTER_KEY
    - SLACK_WEBHOOK_URL
    - GOOGLE_CLIENT_ID
    - GOOGLE_CLIENT_SECRET
    - CLOUDFRONT_ENDPOINT
    - POSTGRES_PASSWORD

Don't forget to add your environment variables as well.

We need to update config/database.yml file to match our configuration in deploy.yml:

default: &default
  adapter: postgresql
  encoding: unicode
  # For details on connection pooling, see Rails configuration guide
  # https://guides.rubyonrails.org/configuring.html#database-pooling
  pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>

production:
  <<: *default
  username: <%= ENV["POSTGRES_USER"] %>
  password: <%= ENV["POSTGRES_PASSWORD"] %>
  database: <%= ENV["POSTGRES_DB"] %>
  host: <%= ENV["POSTGRES_HOST"] %> # Because we don't use DATABASE_URL, we need to add this line.

And use Redis as cache store in config/environments/production.rb:

config.cache_store = :redis_cache_store, { url: ENV.fetch('REDIS_URL') }

No need to update Sidekiq, it will use REDIS_URL by default.

3. Update the Dockerfile

In the build stage, you need to add libpq-dev:

FROM base as build

# Install packages needed to build gems and node modules
RUN apt-get update -qq && \
    apt-get install --no-install-recommends -y \
        build-essential \
        curl \
        git \
        libvips \
        node-gyp \
        pkg-config \
        python-is-python3 \
        libpq-dev # This is new

and add postgresql-client is the final stage:

# Install packages needed for deployment
RUN apt-get update -qq && \
    apt-get install --no-install-recommends -y curl postgresql-client && \
    rm -rf /var/lib/apt/lists /var/cache/apt/archives

Note that if you built your application with Rails 7.1 and database externally in case of failure or MySQL directly, it has already been configured for you, you don't have to change anything.

4. Backup to external a host or S3-compatible Object Storage service

It is crucial to properly back up our database externally in case of failure. Thanks to our Kamal-deployed infrastructure, we are not tied to a specific server provider, making migration so much easier!

Even if we use Docker volume on our db accessory, we can not just copy the folder and send it on another server. We need to build a proper dump with the command pg_dump that will make consistent backups even if the database is being used concurrently.

In my case, I will host the dump on a S3-compatible Object Storage service called Object Storage on Scaleway, but it can really be anywhere even on the cheapest and dumbest server using rsync, for example. It just has to be another server than the one your database is running on.

I will use the Docker image from eeshugerman/postgres-backup-s3 to dump and upload it on Object Storage.

ℹ️ You might be not comfortable using third-party scripts with your database and S3 credentials, and in this case I suggest you to write your own bash scripts. It's really just basic commands that dump and upload files in a crontab.

accessories:
  # ...
  
  s3_backup:
    image: eeshugerman/postgres-backup-s3:16
    host: 192.168.0.1
    env:
      clear:
        SCHEDULE: '@daily'
        BACKUP_KEEP_DAYS: 30
        S3_REGION: your-s3-region
        S3_BUCKET: your-s3-bucket
        S3_PREFIX: backups
        S3_ENDPOINT: https://your-s3-endpoint
        POSTGRES_HOST: my_awesome_app-db
        POSTGRES_DATABASE: my_awesome_app_production
        POSTGRES_USER: my_awesome_app
      secret:
        - POSTGRES_PASSWORD
        - S3_ACCESS_KEY_ID
        - S3_SECRET_ACCESS_KEY
    options:
      network: "private" # This is important!

Don't forget to add the environment variables in your .env and adapt other variables according to your S3 configuration.

5. Healthcheck

By default, a health check request will occur every single second on your app to ensure that Traefik can send requests to it. Depending on your needs, you might want to change the frequency:

healthcheck:
  interval: 5s

6. Put everything together

Let's put it all together and we should be good to go

See the complete deploy.yml: 
# Name of your application. Used to uniquely configure containers.
service: my_awesome_app

# Name of the container image.
image: my_awesome_app

# Deploy to these servers.
servers:
  web:
    hosts:
      - 192.168.0.1
    labels:
      traefik.http.routers.my_awesome_app.entrypoints: websecure
      traefik.http.routers.my_awesome_app.rule: Host(`my_awesome_app.com`)
      traefik.http.routers.my_awesome_app.tls.certresolver: letsencrypt
    options:
      network: "private"

  job:
    hosts:
      - 192.168.0.1
    cmd: bundle exec sidekiq -q default -q mailers
    options:
      network: "private"

# Credentials for your image host.
registry:
  # Specify the registry server, if you're not using Docker Hub
  # server: registry.digitalocean.com / ghcr.io / ...
  username: guillaumebriday

  # Always use an access token rather than real password when possible.
  password:
    - KAMAL_REGISTRY_PASSWORD # Must be present in your `.env`.

# Inject ENV variables into containers (secrets come from .env).
# Remember to run `kamal env push` after making changes!
env:
  clear:
    RAILS_SERVE_STATIC_FILES: true
    POSTGRES_USER: "my_awesome_app"
    POSTGRES_DB: "my_awesome_app_production"
    POSTGRES_HOST: "my_awesome_app-db"
    REDIS_URL: "redis://my_awesome_app-redis:6379/0"
  secret:
    - RAILS_MASTER_KEY
    - SLACK_WEBHOOK_URL
    - GOOGLE_CLIENT_ID
    - GOOGLE_CLIENT_SECRET
    - CLOUDFRONT_ENDPOINT
    - POSTGRES_PASSWORD

# Use accessory services (secrets come from .env).
accessories:
  db:
    image: postgres:16
    host: 192.168.0.1
    env:
      clear:
        POSTGRES_USER: "my_awesome_app"
        POSTGRES_DB: "my_awesome_app_production"
      secret:
        - POSTGRES_PASSWORD
    directories:
      - data:/var/lib/postgresql/data
    options:
      network: "private"

  redis:
    image: redis:7.0
    host: 192.168.0.1
    directories:
      - data:/data
    options:
      network: "private"

  s3_backup:
    image: eeshugerman/postgres-backup-s3:16
    host: 192.168.0.1
    env:
      clear:
        SCHEDULE: '@daily'
        BACKUP_KEEP_DAYS: 30
        S3_REGION: your-s3-region
        S3_BUCKET: your-s3-bucket
        S3_PREFIX: backups
        S3_ENDPOINT: https://your-s3-endpoint
        POSTGRES_HOST: my_awesome_app-db
        POSTGRES_DATABASE: my_awesome_app_production
        POSTGRES_USER: my_awesome_app
      secret:
        - POSTGRES_PASSWORD
        - S3_ACCESS_KEY_ID
        - S3_SECRET_ACCESS_KEY
    options:
      network: "private"

# Configure custom arguments for Traefik
traefik:
  options:
    publish:
      - "443:443"
    volume:
      - "/letsencrypt/acme.json:/letsencrypt/acme.json" # To save the configuration file.
    network: "private" 
  args:
    entryPoints.web.address: ":80"
    entryPoints.websecure.address: ":443"
    entryPoints.web.http.redirections.entryPoint.to: websecure # We want to force https
    entryPoints.web.http.redirections.entryPoint.scheme: https
    entryPoints.web.http.redirections.entrypoint.permanent: true
    certificatesResolvers.letsencrypt.acme.email: "email@example.com"
    certificatesResolvers.letsencrypt.acme.storage: "/letsencrypt/acme.json" 
    certificatesResolvers.letsencrypt.acme.httpchallenge: true
    certificatesResolvers.letsencrypt.acme.httpchallenge.entrypoint: web

healthcheck:
  interval: 5s

7. Conclusion

It's easy as that, enjoy your reliable and cheap deploy! 🚀

📢 This post is part of a series on Kamal

  1. Easy, Perfect for side/small projects: How to deploy Rails with Kamal and SSL certificate on any VPS
  2. Medium: Perfect for most projects: How to deploy Rails with Kamal, PostgreSQL, Sidekiq and Backups on a single host
  3. Expert: Perfect for big projects: How to Deploy and Scale your Rails application with Kamal

Simplify your time tracking with Timecop

Timecop is a time tracking app that brings simplicity in your day to day life.

Get startedLearn more
Timecop projects